plugins/elements/stack.py: Require all dependencies be build & run.

Stack elements cannot be build-only dependencies, as this would defeat
the purpose of using stack elements in order to directly build-depend on
them.

Stack element dependencies must all be built in order to build depend
on them, and as such we gain no build parallelism by allowing runtime-only
dependencies on stack elements. Declaring a runtime-only dependency on
a stack element as a whole might still be useful, but still requires the
entire stack to be built at the time we need that stack.

Instead, it is more useful to ensure that a stack element is a logical
group of all dependencies, including runtime dependencies, such that we
can guarantee cache key alignment with all stack dependencies.

This allows for stronger reliability in commands such as
`bst artifact checkout`, which can now reliably download and checkout
a fully built stack as a result, without any uncertainty about possible
runtime-only dependencies which might exist in the project where that
artifact was created.

This consequently closes #1075

This also fixes the following tests such that the no longer
require build-depends or runtime-depends to work in stack elements:

 * tests/frontend/default_target.py: Was not necessary to check results of show,
   these stacks were set to runtime-depends so that they would have the same
   buildable state as their dependencies when shown.

 * tests/format/dependencies.py: tests/frontend/pull.py, test/frontend/show.py,
   tests/integration/compose.py:
   These tests were using specific build/runtime dependencies in stacks, but
   for no particular reason.

10 files changed, 126 insertions, 33 deletions

diff --git a/src/buildstream/plugins/elements/stack.py b/src/buildstream/plugins/elements/stack.py
index b15f67073..bd914ed0a 100644
--- a/src/buildstream/plugins/elements/stack.py
+++ b/src/buildstream/plugins/elements/stack.py
@@ -1,5 +1,5 @@
 #
-#  Copyright (C) 2016 Codethink Limited
+#  Copyright (C) 2020 Codethink Limited
 #
 #  This program is free software; you can redistribute it and/or
 #  modify it under the terms of the GNU Lesser General Public
@@ -22,9 +22,85 @@ stack - Symbolic Element for dependency grouping
 ================================================
 Stack elements are simply a symbolic element used for representing
 a logical group of elements.
+
+All dependencies declared in stack elements must always be both
+:ref:`build and runtime dependencies <format_dependencies_types>`.
+
+**Example:**
+
+.. code:: yaml
+
+   kind: stack
+
+   # Declare all of your dependencies in the `depends` list.
+   depends:
+   - libc.bst
+   - coreutils.bst
+
+.. note::
+
+   Unlike other elements, whose cache keys are a unique identifier
+   of the contents of the artifacts they produce, stack elements do
+   not produce any artifact content. Instead, the cache key of an artifact
+   is a unique identifier for the assembly of its own dependencies.
+
+
+Using intermediate stacks
+-------------------------
+Using a stack element at intermediate levels of your build graph
+allows you to abstract away some parts of your project into logical
+subsystems which elements can more conveniently depend on as a whole.
+
+In addition to the added convenience, it will allow you to more
+easily change the implementation of a subsystem later on, without needing
+to update many reverse dependencies to depend on new elements, or even
+allow you to conditionally implement a subsystem with various implementations
+depending on what :ref:`project options <project_options>` were specified at
+build time.
+
+
+Using toplevel stacks
+---------------------
+Stack elements can also be useful as toplevel targets in your build graph
+to simply indicate all of the components which need to be built for a given
+system to be complete, or for your integration pipeline to be successful.
+
+
+Checking out and deploying toplevel stacks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+In case that your software is built remotely, it is possible to checkout
+the built content of a stack on your own machine for the purposes of
+inspection or further deployment.
+
+To accomplish this, you will need to know the cache key of the stack element
+which was built remotely, possibly by inspecting the remote build log or by
+deriving it with an equally configured BuildStream project, and you will
+need read access to the artifact cache server which the build was uploaded to,
+this should be configured in your :ref:`user configuration file <config_artifacts>`.
+
+You can then checkout the remotely built stack using the
+:ref:`bst artifact checkout <invoking_artifact_checkout>` command and providing
+it with the :ref:`artifact name <artifact_names>`:
+
+**Example:**
+
+.. code:: shell
+
+   bst artifact checkout --deps build --pull --integrate \\
+       --directory `pwd`/checkout \\
+       project/stack/788da21e7c1b5818b7e7b60f7eb75841057ff7e45d362cc223336c606fe47f27
+
+.. note::
+
+   It is possible to checkout other elements in the same way, however stack
+   elements are uniquely suited to this purpose, as they cannot have
+   :ref:`runtime only dependencies <format_dependencies_types>`, and consequently
+   their cache keys are always a unique representation of their collective
+   dependencies.
 """
 
-from buildstream import Element
+from buildstream import Element, ElementError
+from buildstream.types import _Scope
 
 
 # Element implementation for the 'stack' kind.
@@ -46,7 +122,20 @@ class StackElement(Element):
         pass
 
     def preflight(self):
-        pass
+
+        # Assert that all dependencies are both build and runtime dependencies.
+        #
+        all_deps = list(self._dependencies(_Scope.ALL, recurse=False))
+        run_deps = list(self._dependencies(_Scope.RUN, recurse=False))
+        build_deps = list(self._dependencies(_Scope.BUILD, recurse=False))
+        if any(dep not in run_deps for dep in all_deps) or any(dep not in build_deps for dep in all_deps):
+            # There is no need to specify the `self` provenance here in preflight() errors, as the base class
+            # will take care of prefixing these for plugin author convenience.
+            raise ElementError(
+                "All dependencies of 'stack' elements must be both build and runtime dependencies",
+                detail="Make sure you declare all dependencies in the `depends` list, without specifying any `type`.",
+                reason="stack-requires-build-and-run",
+            )
 
     def get_unique_key(self):
         # We do not add anything to the build, only our dependencies
diff --git a/tests/format/dependencies1/elements/builddep-list.bst b/tests/format/dependencies1/elements/builddep-list.bst
index a0cbcaf23..eb216a127 100644
--- a/tests/format/dependencies1/elements/builddep-list.bst
+++ b/tests/format/dependencies1/elements/builddep-list.bst
@@ -1,4 +1,4 @@
-kind: stack
+kind: manual
 description: This element has a build-only dependency specified via build-depends
 build-depends:
-  - firstdep.bst
+- firstdep.bst
diff --git a/tests/format/dependencies1/elements/list-combine.bst b/tests/format/dependencies1/elements/list-combine.bst
index ed3452206..d39ddd38b 100644
--- a/tests/format/dependencies1/elements/list-combine.bst
+++ b/tests/format/dependencies1/elements/list-combine.bst
@@ -1,4 +1,4 @@
-kind: stack
+kind: manual
 description: This element depends on three elements in different ways
 build-depends:
 - firstdep.bst
diff --git a/tests/format/dependencies1/elements/runtimedep-list.bst b/tests/format/dependencies1/elements/runtimedep-list.bst
index 1207a492d..eaa0cd2e0 100644
--- a/tests/format/dependencies1/elements/runtimedep-list.bst
+++ b/tests/format/dependencies1/elements/runtimedep-list.bst
@@ -1,4 +1,4 @@
-kind: stack
+kind: manual
 description: This element has a runtime-only dependency
 runtime-depends:
-  - firstdep.bst
+- firstdep.bst
diff --git a/tests/frontend/default-target/elements/dummy_stack.bst b/tests/frontend/default-target/elements/dummy_stack.bst
index 5f921667f..3ea51b00f 100644
--- a/tests/frontend/default-target/elements/dummy_stack.bst
+++ b/tests/frontend/default-target/elements/dummy_stack.bst
@@ -1,5 +1,5 @@
 kind: stack
 
-runtime-depends:
+depends:
 - dummy_1.bst
 - dummy_2.bst
diff --git a/tests/frontend/default_target.py b/tests/frontend/default_target.py
index bb7a49592..60578bb8e 100644
--- a/tests/frontend/default_target.py
+++ b/tests/frontend/default_target.py
@@ -13,27 +13,27 @@ from tests.testutils import create_artifact_share
 DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "default-target",)
 
 
-###################################
-#      build/show operations      #
-###################################
-
-
+#
+# When no target is default, then expect all targets to be built
+#
 @pytest.mark.datafiles(DATA_DIR)
-@pytest.mark.parametrize("operation,expected_state", [("show", "buildable"), ("build", "cached")])
-def test_no_default(cli, datafiles, operation, expected_state):
+def test_no_default(cli, datafiles):
     project = str(datafiles)
     all_targets = ["dummy_1.bst", "dummy_2.bst", "dummy_3.bst", "dummy_stack.bst"]
 
-    result = cli.run(project=project, args=[operation])
+    result = cli.run(project=project, args=["build"])
     result.assert_success()
 
     states = cli.get_element_states(project, all_targets)
-    assert all(states[e] == expected_state for e in all_targets)
+    assert all(states[e] == "cached" for e in all_targets)
 
 
+#
+# When the stack is specified as the default target, then
+# expect only it and it's dependencies to be built
+#
 @pytest.mark.datafiles(DATA_DIR)
-@pytest.mark.parametrize("operation,expected_state", [("show", "buildable"), ("build", "cached")])
-def test_default_target(cli, datafiles, operation, expected_state):
+def test_default_target(cli, datafiles):
     project = str(datafiles)
     project_path = os.path.join(project, "project.conf")
 
@@ -49,19 +49,23 @@ def test_default_target(cli, datafiles, operation, expected_state):
     # dummy_stack only depends on dummy_1 and dummy_2, but not dummy_3
     all_targets = ["dummy_1.bst", "dummy_2.bst", "dummy_stack.bst"]
 
-    result = cli.run(project=project, args=[operation])
+    result = cli.run(project=project, args=["build"])
     result.assert_success()
 
     states = cli.get_element_states(project, all_targets)
-    assert all(states[e] == expected_state for e in all_targets)
+    assert all(states[e] == "cached" for e in all_targets)
 
     # assert that dummy_3 isn't included in the output
     assert "dummy_3.bst" not in states
 
 
+#
+# Even when there is a junction, expect that the elements in the
+# subproject referred to by the toplevel project are built when
+# calling `bst build` and no default is specified.
+#
 @pytest.mark.datafiles(DATA_DIR)
-@pytest.mark.parametrize("operation,expected_state", [("show", "buildable"), ("build", "cached")])
-def test_no_default_with_junction(cli, datafiles, operation, expected_state):
+def test_no_default_with_junction(cli, datafiles):
     project = str(datafiles)
     junction_path = os.path.join(project, "elements", "junction.bst")
     target_path = os.path.join(project, "elements", "junction-target.bst")
@@ -71,16 +75,16 @@ def test_no_default_with_junction(cli, datafiles, operation, expected_state):
     _yaml.roundtrip_dump(junction_config, junction_path)
 
     # Then, create a stack element with dependency on cross junction element
-    target_config = {"kind": "stack", "runtime-depends": ["junction.bst:dummy_subproject.bst"]}
+    target_config = {"kind": "stack", "depends": ["junction.bst:dummy_subproject.bst"]}
     _yaml.roundtrip_dump(target_config, target_path)
 
-    # Now try to perform the specified operation.
+    # Now try to perform a build
     # This should automatically fetch the junction at load time.
-    result = cli.run(project=project, args=[operation])
+    result = cli.run(project=project, args=["build"])
     result.assert_success()
 
-    assert cli.get_element_state(project, "junction.bst:dummy_subproject.bst") == expected_state
-    assert cli.get_element_state(project, "junction-target.bst") == expected_state
+    assert cli.get_element_state(project, "junction.bst:dummy_subproject.bst") == "cached"
+    assert cli.get_element_state(project, "junction-target.bst") == "cached"
 
 
 ###################################
diff --git a/tests/frontend/pull.py b/tests/frontend/pull.py
index c1fa76ab0..f873ad96e 100644
--- a/tests/frontend/pull.py
+++ b/tests/frontend/pull.py
@@ -338,7 +338,7 @@ def test_pull_missing_local_blob(cli, tmpdir, datafiles):
     _yaml.roundtrip_dump(input_config, input_file)
 
     depends_name = "depends.bst"
-    depends_config = {"kind": "stack", "depends": [{"filename": input_name, "type": "build"}]}
+    depends_config = {"kind": "stack", "depends": [input_name]}
     depends_file = os.path.join(element_dir, depends_name)
     _yaml.roundtrip_dump(depends_config, depends_file)
 
diff --git a/tests/frontend/strict-depends/elements/non-strict-depends.bst b/tests/frontend/strict-depends/elements/non-strict-depends.bst
index 9ab119bde..e0ed21ed6 100644
--- a/tests/frontend/strict-depends/elements/non-strict-depends.bst
+++ b/tests/frontend/strict-depends/elements/non-strict-depends.bst
@@ -1,4 +1,4 @@
 kind: stack
 
-build-depends:
+depends:
 - base.bst
diff --git a/tests/frontend/strict-depends/elements/strict-depends.bst b/tests/frontend/strict-depends/elements/strict-depends.bst
index 1e4e29414..10f3b9418 100644
--- a/tests/frontend/strict-depends/elements/strict-depends.bst
+++ b/tests/frontend/strict-depends/elements/strict-depends.bst
@@ -1,5 +1,5 @@
 kind: stack
 
-build-depends:
+depends:
 - filename: base.bst
   strict: true
diff --git a/tests/integration/project/elements/compose/test-integration.bst b/tests/integration/project/elements/compose/test-integration.bst
index 2f9faf170..ef47b3b58 100644
--- a/tests/integration/project/elements/compose/test-integration.bst
+++ b/tests/integration/project/elements/compose/test-integration.bst
@@ -1,6 +1,6 @@
 kind: stack
 
-runtime-depends:
+depends:
 - base.bst
 
 public: