doc/examples/composition: Adding user guide about split-rules and compose elementstristan/user-guide-refactor-test

1 files changed, 194 insertions, 0 deletions

diff --git a/doc/source/handling-files/composition.rst b/doc/source/handling-files/composition.rst
new file mode 100644
index 000000000..729ed88ee
--- /dev/null
+++ b/doc/source/handling-files/composition.rst
@@ -0,0 +1,194 @@
+
+
+.. _handling_files_composition:
+
+Composition
+===========
+In this chapter we will explore how to create *compositions* of multiple
+input filesystem trees, using the :mod:`compose <elements.compose>` element.
+
+.. note::
+
+   This example is distributed with BuildStream
+   in the `doc/examples/composition
+   <https://gitlab.com/BuildStream/buildstream/tree/master/doc/examples/composition>`_
+   subdirectory.
+
+
+Overview
+--------
+Composing a directory tree based on a set of build dependencies is often
+one of the important steps you might perform in order to create a single artifact
+which can be checked out and deployed.
+
+In order to use the :mod:`compose <elements.compose>` element, it is important
+to first understand the concept of :ref:`split rules <public_split_rules>`, which
+we will cover in this chapter.
+
+
+Introducing split rules
+~~~~~~~~~~~~~~~~~~~~~~~
+The :ref:`split rules <public_split_rules>` of an element declaration denote
+which sets of files in the given element's resulting artifact belong to which
+*domain name*.
+
+The *domains* can then be used in various ways, using plugins which understand
+*split rule domains*.
+
+BuildStream's :ref:`default project configuration <project_builtin_defaults>`
+contains a sensible set of default *split rule domains* for the purpose of
+artifact splitting, they can be overridden in :ref:`your project.conf <project_split_rules>`,
+and finally on a per element basis in the :ref:`public data <public_builtin>`
+of your element declarations.
+
+.. note::
+
+   Projects are free to add additional *split rule domains* on top of the
+   default domains provided by the default project configuration.
+
+   There is nothing wrong with defining split rule domains which *overlap*,
+   possibly capturing some of the same files also captured by another
+   *domain*, however you should be aware of this when later using your
+   split rules with a plugin which processes them, like the
+   :mod:`compose <elements.compose>` element described in this chapter.
+
+
+Example of split rule declaration
+'''''''''''''''''''''''''''''''''
+In an element, you might need to define or extend the ``split-rules``
+in order to capture files in custom locations in a logical *domain*.
+
+Here is an example of how you might use the
+:ref:`list append directive <format_directives_list_append>`
+to append an additional rule to your ``split-rules`` list in order to
+capture additional data files which your application or library might
+want to include in the *runtime domain*:
+
+.. code:: yaml
+
+   # Add our .dat files to the runtime domain
+   public:
+     bst:
+       split-rules:
+         runtime:
+	   (>):
+           - |
+             %{datadir}/foo/*.dat
+
+Split rules are absolute paths which denote files within an artifact's root
+directory. The globbing patterns supported in split rules are defined in the
+:func:`reference documentation here <buildstream.utils.glob>`.
+
+.. important::
+
+   Note that because of variable expansion, split rules can often be
+   *resolved differently* for elements which have overridden path
+   related variables, like ``%{prefix}``.
+
+   This usually means that you do not need to explicitly extend or override
+   split rules on a specific element unless your element installs files to
+   special case locations.
+
+
+Project structure
+-----------------
+In this example we expand on the chapter about
+:ref:`integration commands <tutorial_integration_commands>`, so we will
+only discuss the files which are added or changed from that example.
+
+
+``elements/base/alpine.bst``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. literalinclude:: ../../examples/composition/elements/base/alpine.bst
+   :language: yaml
+
+Here we have modified the base runtime, so as to specify that for this
+element, we want to also include the runtime linker into the *runtime domain*.
+
+
+``elements/runtime-only.bst``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. literalinclude:: ../../examples/composition/elements/runtime-only.bst
+   :language: yaml
+
+As we can see, this :mod:`compose <elements.compose>` element has
+been configured to only include files from the *runtime domain*.
+
+
+Using the project
+-----------------
+Now that we've presented how :ref:`split rules <public_split_rules>`
+work and shown how to use them in the context of this example, lets
+use the :mod:`compose <elements.compose>` element we've created and
+observe the results.
+
+
+Building the project
+~~~~~~~~~~~~~~~~~~~~
+
+.. raw:: html
+   :file: ../sessions/composition-build.html
+
+As you can see in the output, this composition has only a few hundred
+files, but the complete ``alpine.bst`` runtime has several thousand
+files.
+
+
+List the content
+~~~~~~~~~~~~~~~~
+At the risk of this being a long list, let's :ref:`list the
+contents of this artifact <invoking_artifact_list_contents>`
+
+.. raw:: html
+   :file: ../sessions/composition-list-contents.html
+
+Some things to observe here:
+
+* The list does include the ``/usr/bin/hello`` program and
+  also the ``/usr/lib/libhello.so`` shared library.
+
+  These paths are both captured by the default split rules
+  for the *runtime domain*.
+
+* The list does not include the ``/usr/include/libhello.h``
+  header file which was used to compile ``/usr/bin/hello``.
+
+  The header file is not captured by the *runtime domain*
+  by default. It is however captured by the *devel domain*.
+
+* The runtime linker ``/lib/ld-musl-x86_64.so.1``, as this was
+  explicitly added to the *runtime domain* for the ``base/alpine.bst``
+  element which provides this file.
+
+.. tip::
+
+   The reader at this time might want to list the content of
+   other elements built from this project, such as the
+   ``hello.bst`` element by itself, or the ``base/alpine.bst``
+   element.
+
+
+Run the program
+~~~~~~~~~~~~~~~
+Finally, lets just run the program we built.
+
+.. raw:: html
+   :file: ../sessions/composition-shell.html
+
+Here we can see that we at least have the required files to run
+our hello world program, however we would not have if we were
+missing the runtime linker which we added in ``base/alpine.bst``.
+
+
+Summary
+-------
+In this chapter we've gotten familiar with :ref:`split rules <public_split_rules>`
+annotations, and we've learned enough about the :mod:`compose <elements.compose>`
+element such that we can start creating our own compositions using
+*split domains*.
+
+We've also used the :ref:`list append directive <format_directives_list_append>`
+and we are now observing the contents of artifacts using
+:ref:`bst artifact list-contents <invoking_artifact_list_contents>`.