doc/examples/junction-includes: Adding new example about includes with junctions

11 files changed, 376 insertions, 0 deletions

diff --git a/doc/examples/junction-includes/elements/hello.bst b/doc/examples/junction-includes/elements/hello.bst
new file mode 100644
index 000000000..bdfd1f051
--- /dev/null
+++ b/doc/examples/junction-includes/elements/hello.bst
@@ -0,0 +1,23 @@
+kind: autotools
+description: |
+
+    Hello world example from automake
+
+variables:
+
+  # The special paths.bst from our subproject is used to
+  # define the paths of some elements in this project.
+  #
+  (@): subproject-junction.bst:include/paths.bst
+
+  # The hello world example lives in the doc/amhello folder.
+  command-subdir: doc/amhello
+
+sources:
+- kind: tar
+  url: gnu:automake-1.16.tar.gz
+  ref: 80da43bb5665596ee389e6d8b64b4f122ea4b92a685b1dbd813cd1f0e0c2d83f
+
+depends:
+  - filename: base.bst
+    junction: subproject-junction.bst
diff --git a/doc/examples/junction-includes/elements/subproject-junction.bst b/doc/examples/junction-includes/elements/subproject-junction.bst
new file mode 100644
index 000000000..0df4e7646
--- /dev/null
+++ b/doc/examples/junction-includes/elements/subproject-junction.bst
@@ -0,0 +1,17 @@
+kind: junction
+
+config:
+  # Configure the options for subproject
+  #
+  # If our project is funky, then it requires
+  # a blue subproject, otherwise we use a red one.
+  #
+  options:
+    color: red
+    (?):
+    - funky == True:
+        color: blue
+
+sources:
+- kind: local
+  path: subproject
diff --git a/doc/examples/junction-includes/project.conf b/doc/examples/junction-includes/project.conf
new file mode 100644
index 000000000..e591b78b1
--- /dev/null
+++ b/doc/examples/junction-includes/project.conf
@@ -0,0 +1,14 @@
+name: subproject
+format-version: 18
+element-path: elements
+
+aliases:
+  gnu: http://ftpmirror.gnu.org/gnu/automake/
+
+# Define some options for this project
+#
+options:
+  funky:
+    type: bool
+    description: Whether this project is funky
+    default: False
diff --git a/doc/examples/junction-includes/subproject/elements/base.bst b/doc/examples/junction-includes/subproject/elements/base.bst
new file mode 100644
index 000000000..1b85a9e8c
--- /dev/null
+++ b/doc/examples/junction-includes/subproject/elements/base.bst
@@ -0,0 +1,5 @@
+kind: stack
+description: Base stack
+
+depends:
+- base/alpine.bst
diff --git a/doc/examples/junction-includes/subproject/elements/base/alpine.bst b/doc/examples/junction-includes/subproject/elements/base/alpine.bst
new file mode 100644
index 000000000..cf85df5bf
--- /dev/null
+++ b/doc/examples/junction-includes/subproject/elements/base/alpine.bst
@@ -0,0 +1,13 @@
+kind: import
+description: |
+
+    Alpine Linux base runtime
+
+sources:
+- kind: tar
+
+  # This is a post doctored, trimmed down system image
+  # of the Alpine linux distribution.
+  #
+  url: alpine:integration-tests-base.v1.x86_64.tar.xz
+  ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639
diff --git a/doc/examples/junction-includes/subproject/include/paths.bst b/doc/examples/junction-includes/subproject/include/paths.bst
new file mode 100644
index 000000000..b9ab2e638
--- /dev/null
+++ b/doc/examples/junction-includes/subproject/include/paths.bst
@@ -0,0 +1,7 @@
+# When this project color is blue, including this
+# file causes installations to be relocated to /opt
+#
+prefix: /usr
+(?):
+- color == "blue":
+    prefix: /opt
diff --git a/doc/examples/junction-includes/subproject/project.conf b/doc/examples/junction-includes/subproject/project.conf
new file mode 100644
index 000000000..d9387f244
--- /dev/null
+++ b/doc/examples/junction-includes/subproject/project.conf
@@ -0,0 +1,18 @@
+name: subproject
+format-version: 18
+element-path: elements
+
+aliases:
+  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
+
+# Define some options for this project
+#
+options:
+  color:
+    type: enum
+    description: The color of this runtime
+    values:
+    - red
+    - green
+    - blue
+    default: blue
diff --git a/doc/sessions/junction-includes.run b/doc/sessions/junction-includes.run
new file mode 100644
index 000000000..737beb33a
--- /dev/null
+++ b/doc/sessions/junction-includes.run
@@ -0,0 +1,25 @@
+
+commands:
+# Make it fetch first
+- directory: ../examples/junction-includes
+  command: source fetch hello.bst
+
+# Build hello.bst normally
+- directory: ../examples/junction-includes
+  output: ../source/sessions/junction-includes-build-normal.html
+  command: build hello.bst
+
+# Build hello.bst funky
+- directory: ../examples/junction-includes
+  output: ../source/sessions/junction-includes-build-funky.html
+  command: --option funky True build hello.bst
+
+# Run hello.bst in a shell
+- directory: ../examples/junction-includes
+  output: ../source/sessions/junction-includes-shell-normal.html
+  command: shell hello.bst -- /usr/bin/hello
+
+# Run hello.bst in a shell
+- directory: ../examples/junction-includes
+  output: ../source/sessions/junction-includes-shell-funky.html
+  command: --option funky True shell hello.bst -- /opt/bin/hello
diff --git a/doc/source/junctions/junction-includes.rst b/doc/source/junctions/junction-includes.rst
new file mode 100644
index 000000000..17de81b79
--- /dev/null
+++ b/doc/source/junctions/junction-includes.rst
@@ -0,0 +1,251 @@
+
+
+.. _advanced_junction_includes:
+
+Subproject includes
+===================
+We've already discussed how we can add optionality to projects and
+explored how we can perform conditional statements and include fragments
+of BuildStream YAML in the earlier :ref:`chapter about optionality and
+directives <tutorial_directives>`.
+
+In this chapter we're going to explore how we can use :ref:`include directives
+<format_directives_include>` to include YAML fragments from a subproject
+referred to by a :mod:`junction <elements.junction>` element, and how
+:ref:`project options <project_options>` can be specified in the configuration
+of your :mod:`junction <elements.junction>`.
+
+.. note::
+
+   This example is distributed with BuildStream
+   in the `doc/examples/junction-includes
+   <https://gitlab.com/BuildStream/buildstream/tree/master/doc/examples/junction-includes>`_
+   subdirectory.
+
+
+Overview
+--------
+It is a goal of BuildStream to provide developers and integrators with the tools
+they need to maintain software stacks which depend on eachother with least friction
+as possible, such that one can integrate upgrades of projects one depends on
+via :mod:`junction <elements.junction>` elements regularly and with the least
+hassle as possible.
+
+:ref:`Project options <project_options>` and :ref:`include directives
+<format_directives_include>` combined form the basis on which projects can
+maximize on code sharing effectively, and the basis on which BuildStream
+projects can form reliable APIs.
+
+
+Project options
+~~~~~~~~~~~~~~~
+The :ref:`options <project_options>` which a project exposes is a fairly
+limited API surface, it allows one to configure a limited set of options
+advertized by the project maintainers, and the options will affect what
+kind of artifacts will be produced by the project.
+
+This kind of optionality however does not allow consumers to entirely
+redefine how artifacts are produced and how elements are configured.
+
+On the one hand, this limitation can be frustrating, as one constantly
+finds themselves requiring a feature that their subproject does not
+support *right now*. On the other hand, the limitation of features which
+a given project chooses to support is what guards downstream project
+consumers against consuming artifacts which are not supported by the upstream.
+
+Project options are designed to enforce a *separation of concerns*,
+where we expect that downstreams will either fork a project in order
+to support a new feature, or convince the upstream to start supporting
+a new feature. Furthermore, limited API surfaces for interdependent
+projects offers a possibility of API stability of projects, such
+that you can upgrade your dependencies with limited friction.
+
+
+Includes
+~~~~~~~~
+The :ref:`includes <format_directives_include>` which a project might advertize
+as *"public"*, form the output of the API exchange between a project and
+its subproject(s).
+
+Cross-project include files allow a project to *inherit configuration* from
+a subproject. Include files can be used to define anything from the
+:ref:`variables <format_variables>` one needs to have in context in order to
+build into or link into alternative system prefixes, what special compiler flags
+to use when building for a specific machine architecture, to customized
+:ref:`shell configurations <project_shell>` to use when testing out applications
+in :ref:`bst shell <invoking_shell>`.
+
+This chapter will provide an example of the *mechanics* of cross project
+includes when combined with project optionality.
+
+
+Project structure
+-----------------
+
+
+Project options
+~~~~~~~~~~~~~~~
+This example is comprised of two separate projects, both of which offer
+some project options. This is intended to emphasize how your toplevel project
+options can be used to select and configure options to use in the subprojects
+you depend on.
+
+For convenience, the subproject is stored in the subdirectory of
+the toplevel project, while in the real world the subproject is probably
+hosted elsewhere.
+
+First let's take a look at the options declared in the respective
+``project.conf`` files.
+
+
+Toplevel ``project.conf``
+'''''''''''''''''''''''''
+
+.. literalinclude:: ../../examples/junction-includes/project.conf
+   :language: yaml
+
+
+
+Subproject ``project.conf``
+'''''''''''''''''''''''''''
+
+.. literalinclude:: ../../examples/junction-includes/subproject/project.conf
+   :language: yaml
+
+
+As we can see, these two projects both offer some arbitrarily named options.
+
+
+Conditional configuration of subproject
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The toplevel project here does some conditional configuration of the
+subproject.
+
+
+Toplevel ``elements/subproject-junction.bst``
+'''''''''''''''''''''''''''''''''''''''''''''
+
+.. literalinclude:: ../../examples/junction-includes/elements/subproject-junction.bst
+   :language: yaml
+
+Here we can see that projects can use
+:ref:`conditional statements <format_directives_conditional>` to make
+decisions about subproject configuration based on their own configuration.
+
+In this example, if the toplevel project is ``funky``, then it will
+configure its subproject with ``color`` set to ``blue``, otherwise it
+will use the ``red`` variant of the subproject ``color``.
+
+
+Including configuration from a subproject
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Here there are a couple of aspects to observe, namely how the
+toplevel project includes files across a junction boundary,
+and how that include file might be implemented.
+
+
+Toplevel ``elements/hello.bst``
+'''''''''''''''''''''''''''''''
+
+.. literalinclude:: ../../examples/junction-includes/elements/hello.bst
+   :language: yaml
+
+Here we can see the same element which we discussed in the
+:ref:`autotools example <tutorial_autotools>`, except that we're including
+a file from the subproject. As explained in the :ref:`reference manual <format_directives_include>`,
+this is done by prefixing the include path with the local :mod:`junction <elements.junction>`
+element name and then a colon.
+
+Note that in this case, the API contract is simply that ``hello.bst`` is
+including ``paths.bst``, and has the expectation that ``paths.bst`` will
+in some way influence the ``variables``, nothing more.
+
+It can be that an include file is expected to create new variables, and
+it can be that the subproject might declare things differently depending
+on the subproject's own configuration, as we will observe next.
+
+
+Subproject ``include/paths.bst``
+''''''''''''''''''''''''''''''''
+
+.. literalinclude:: ../../examples/junction-includes/subproject/include/paths.bst
+   :language: yaml
+
+Here, we can see the include file *itself* is making a
+:ref:`conditional statement <format_directives_conditional>`, in turn
+deciding what values to use depending on how the project was configured.
+
+This decision will provide valuable context for any file including ``paths.bst``,
+whether it be an element, a ``project.conf`` which applies the variable as
+a default for the entire project, whether it is being included by files
+in the local project, or whether it is being included by a downstream
+project which junctions this project, as is the case in this example.
+
+
+Using the project
+-----------------
+At this stage, you have probably already reasoned out what would happen
+if we tried to build and run the project.
+
+Nevertheless, we will still present the outputs here for observation.
+
+
+Building the project normally
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Here we build the project without any special arguments.
+
+.. raw:: html
+   :file: ../sessions/junction-includes-build-normal.html
+
+
+Building the project in funky mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Now let's see what happens when we build the project in funky mode
+
+.. raw:: html
+   :file: ../sessions/junction-includes-build-funky.html
+
+As we can see, this time we've built the project into the ``/opt``
+system prefix instead of the standard ``/usr`` prefix.
+
+Let's just take a step back now and summarize the process which
+went into this decision:
+
+* The toplevel ``project.conf`` exposes the boolean ``funky`` option
+
+* The toplevel junction ``subproject-junction.bst`` chooses to set the
+  subproject ``color`` to ``blue`` when the toplevel project is ``funky``
+
+* The subproject ``include/paths.bst`` include file decides to set the
+  ``prefix`` to ``/opt`` in the case that the subproject is ``blue``
+
+* The ``hello.bst`` includes the ``include/paths.bst`` file, in order
+  to inherit its path configuration from the subproject
+
+
+Running the project in both modes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. raw:: html
+   :file: ../sessions/junction-includes-shell-normal.html
+
+.. raw:: html
+   :file: ../sessions/junction-includes-shell-funky.html
+
+As expected, the ``funky`` variant of the toplevel project installs
+the hello world program in the ``/opt`` prefix, and as such we
+need to call it from there.
+
+
+Summary
+-------
+In this chapter we've discussed how :ref:`conditional statements <format_directives_conditional>`
+and :ref:`include files <format_directives_include>` play an essential role
+in the API surface of a project, and help to provide some configurability
+while preserving encapsulation of the API which a project exposes.
+
+We've also gone over the mechanics of how these concepts interact and
+presented an example which shows how project options can be used in
+a recursive context, and how includes can help not only to share code,
+but to provide context to dependent projects about how their subprojects
+are configured.
diff --git a/doc/source/tutorial/autotools.rst b/doc/source/tutorial/autotools.rst
index ea2835838..c8fec3be3 100644
--- a/doc/source/tutorial/autotools.rst
+++ b/doc/source/tutorial/autotools.rst
@@ -1,5 +1,7 @@
 
 
+.. _tutorial_autotools:
+
 Using the autotools element
 ===========================
 In :ref:`the last chapter <tutorial_running_commands>` we observed how the
diff --git a/doc/source/using_junctions.rst b/doc/source/using_junctions.rst
index 85f1638b1..9fe492cb7 100644
--- a/doc/source/using_junctions.rst
+++ b/doc/source/using_junctions.rst
@@ -11,3 +11,4 @@ depend on eachother.
 
    junctions/junction-elements
    junctions/junction-workspaces
+   junctions/junction-includes