docs: Updating documentation regarding junctions and conflicts.tristan/junction-jungle

This commit:

  * Redocuments the `junction.py` element almost in it's entirity

  * Describes the `overrides` feature

  * Describes how to use `link` elements to derive junction configuration
    from subprojects.

  * Adds project.conf documentation for junction `duplicates`

  * Adds project.conf documentation for junction `internal`

2 files changed, 250 insertions, 37 deletions

diff --git a/doc/source/format_project.rst b/doc/source/format_project.rst
index 69c844692..93b7f7812 100644
--- a/doc/source/format_project.rst
+++ b/doc/source/format_project.rst
@@ -220,6 +220,7 @@ and keys, please see: :ref:`Key pair for the server <server_authentication>`.
    new server API. As a result newer buildstream clients won't work with older
    servers.
 
+
 .. _project_essentials_split_artifacts:
 
 Split cache servers
@@ -258,6 +259,7 @@ format for that is as such:
       # currently, also only supported by bst-artifact-server and BuildGrid
       type: both
 
+
 .. _project_source_cache:
 
 Source cache server
@@ -285,6 +287,7 @@ Exactly the same as artifact servers, source cache servers can be specified.
    Source caches also support "splitting" like :ref:`artifact servers
    <project_essentials_split_artifacts>`.
 
+
 .. _project_remote_execution:
 
 Remote execution
@@ -336,6 +339,7 @@ The Remote Execution API can be found via https://github.com/bazelbuild/remote-a
 Remote execution configuration can be also provided in the `user
 configuration <user_config_remote_execution>`.
 
+
 .. _project_essentials_mirrors:
 
 Mirrors
@@ -940,6 +944,100 @@ same syntax as other Flag options.
          enable-debug: True
 
 
+.. _project_junctions:
+
+Junctions
+---------
+In this section of ``project.conf``, we can define the relationship a project
+has with :mod:`junction <elements.junction>` elements in the same project, or
+even in subprojects.
+
+Sometimes when your project has multiple :mod:`junction <elements.junction>` elements,
+a situation can arise where you have multiple instances of the same project loaded
+at the same time. In most cases, you will want to reconcile this conflict by ensuring
+that your projects share the same junction. In order to reconcile conflicts by
+ensuring nested junctions to the same project are shared, please refer to
+:ref:`the documentation on nested junctions <core_junction_nested>`.
+
+In some exceptional cases, it is entirely intentional and appropriate to use
+the same project more than once in the same build pipeline. The attributes
+in the ``junctions`` group here in ``project.conf`` provide some tools you can
+use to explicitly allow the coexistence of the same project multiple times.
+
+
+Duplicate junctions
+~~~~~~~~~~~~~~~~~~~
+In the case that you are faced with an error due to subprojects sharing
+a common sub-subproject, you can use the ``duplicates`` configuration
+in order to allow the said project to be loaded twice.
+
+**Example**:
+
+.. code:: yaml
+
+   junctions:
+
+     duplicates:
+
+       # Here we use the packaging tooling completely separately from
+       # the payload that we are packaging, they are never staged to
+       # the same location in a given sandbox, and as such we would
+       # prefer to allow the 'runtime' project to be loaded separately.
+       #
+       # This statement will ensure that loading the 'runtime' project
+       # from these two locations will not produce any errors.
+       #
+       runtime:
+       - payload.bst:runtime.bst
+       - packaging.bst:runtime.bst
+
+When considering duplicated projects in the same pipeline, all instances
+of the said project need to be marked as ``duplicates`` in order to avoid
+a *conflicting junction error* at load time.
+
+.. tip::
+
+   The declaration of ``duplicates`` is inherited by any dependant projects
+   which may later decide to depend on your project.
+
+   If you depend on a project which itself has ``duplicates``, and you need
+   to duplicate it again, then you only need to declare the new duplicate,
+   you do not need to redeclare duplicates redundantly.
+
+
+Internal junctions
+~~~~~~~~~~~~~~~~~~
+Another way to avoid *conflicting junction errors* when you know that your
+subproject should not conflict with other instances of the same subproject,
+is to declare the said subproject as *internal*.
+
+**Example**:
+
+.. code:: yaml
+
+   junctions:
+
+     # Declare this subproject as "internal" because we know
+     # that we only use it for build dependencies, and as such
+     # we know that it cannot collide with elements in dependant
+     # projects.
+     #
+     internal:
+     - special-compiler.bst
+
+When compared to *duplicates* above, *internal* projects have the advantage
+of never producing any *conflicting junction errors* in dependant projects
+(reverse dependency projects).
+
+This approach is preferrable in cases where you know for sure that dependant
+projects will not be depending directly on elements from your internal
+subproject.
+
+.. attention::
+
+   Declaring a junction as *internal* is a promise that dependant projects
+   will not accrue runtime dependencies on elements in your *internal* subproject.
+
 
 .. _project_defaults:
 
diff --git a/src/buildstream/plugins/elements/junction.py b/src/buildstream/plugins/elements/junction.py
index 3396a04ab..425b917ef 100644
--- a/src/buildstream/plugins/elements/junction.py
+++ b/src/buildstream/plugins/elements/junction.py
@@ -1,5 +1,5 @@
 #
-#  Copyright (C) 2017 Codethink Limited
+#  Copyright (C) 2020 Codethink Limited
 #
 #  This program is free software; you can redistribute it and/or
 #  modify it under the terms of the GNU Lesser General Public
@@ -20,7 +20,7 @@
 """
 junction - Integrate subprojects
 ================================
-This element is a link to another BuildStream project. It allows integration
+This element acts as a window into another BuildStream project. It allows integration
 of multiple projects into a single pipeline.
 
 Overview
@@ -48,6 +48,12 @@ Overview
      # Optionally look in a subpath of the source repository for the project
      path: projects/hello
 
+     # Optionally override junction configurations in the subproject
+     # with a junction declaration in this project.
+     #
+     overrides:
+       subproject-junction.bst: local-junction.bst
+
      # Optionally declare whether elements within the junction project
      # should interact with project remotes (default: False).
      cache-junction-elements: False
@@ -57,11 +63,6 @@ Overview
      # remote(s) (default: False).
      ignore-junction-remotes: False
 
-.. note::
-
-   Junction elements may not specify any dependencies as they are simply
-   links to other projects and are not in the dependency graph on their own.
-
 With a junction element in place, local elements can depend on elements in
 the other BuildStream project using :ref:`element paths <format_element_names>`.
 For example, if you have a ``toolchain.bst`` junction element referring to
@@ -73,41 +74,69 @@ dependency to the compiler like this:
    build-depends:
    - junction: toolchain.bst:gcc.bst
 
-While junctions are elements, only a limited set of element operations is
-supported. They can be tracked and fetched like other elements.
-However, junction elements do not produce any artifacts, which means that
-they cannot be built or staged. It also means that another element cannot
-depend on a junction element itself.
+.. important::
 
-.. note::
+   **Limitations**
 
-   Elements within the subproject are not tracked by default when running
-   `bst source track`. You must specify `--cross-junctions` to the track
-   command to explicitly do it.
+   Junction elements are only connectors which bring multiple projects together,
+   and as such they are not in the element dependency graph. This means that it is
+   illegal to depend on a junction, and it is also illegal for a junction to have
+   dependencies.
+
+   While junctions are elements, a limited set of element operations are
+   supported. Junction elements can be tracked and fetched like other
+   elements but they do not produce any artifacts, which means that they
+   cannot be built or staged.
+
+   Note that when running :ref:`bst source track <invoking_source_track>`
+   on your project, elements found in subprojects are not tracked by default.
+   You may specify ``--cross-junctions`` to the
+   :ref:`bst source track <invoking_source_track>` command to explicitly track
+   elements across junction boundaries.
 
 
 Sources
 -------
-``bst show`` does not implicitly fetch junction sources if they haven't been
-cached yet. However, they can be fetched explicitly:
+The sources of a junction element define how to obtain the BuildStream project
+that the junction connects to.
+
+Most commands, such as :ref:`bst build <invoking_build>`, will automatically
+try to fetch the junction elements required to access any subproject elements which
+are specified as dependencies of the targets provided.
+
+Some commands, such as :ref:`bst show <invoking_show>`, do not do this, and in
+such cases they can be fetched explicitly using
+:ref:`bst source fetch <invoking_source_fetch>`:
 
 .. code::
 
    bst source fetch junction.bst
 
-Other commands such as ``bst build`` implicitly fetch junction sources.
 
 Options
 -------
+Junction elements can configure the :ref:`project options <project_options>`
+in the subproject, using the ``options`` configuration.
+
 .. code:: yaml
 
-   options:
-     machine_arch: "%{machine_arch}"
-     debug: True
+   kind: junction
+
+   ...
+
+   config:
+
+     # Specify the options for this subproject
+     #
+     options:
+       machine_arch: "%{machine_arch}"
+       debug: True
+
+Options are never implicitly propagated across junctions, however
+:ref:`variables <format_variables>` can be used to explicitly assign
+configuration in a subproject which matches the toplevel project's
+configuration.
 
-Junctions can configure options of the linked project. Options are never
-implicitly inherited across junctions, however, variables can be used to
-explicitly assign the same value to a subproject option.
 
 .. _core_junction_nested:
 
@@ -116,33 +145,119 @@ Nested Junctions
 Junctions can be nested. That is, subprojects are allowed to have junctions on
 their own. Nested junctions in different subprojects may point to the same
 project, however, in most use cases the same project should be loaded only once.
-BuildStream uses the junction element name as key to determine which junctions
-to merge. It is recommended that the name of a junction is set to the same as
-the name of the linked project.
 
 As the junctions may differ in source version and options, BuildStream cannot
 simply use one junction and ignore the others. Due to this, BuildStream requires
-the user to resolve possibly conflicting nested junctions by creating a junction
-with the same name in the top-level project, which then takes precedence.
+the user to resolve conflicting nested junctions, and will provide an error
+message whenever a conflict is detected.
+
+
+Overriding subproject junctions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+If your project and a subproject share a subproject in common, then one way
+to resolve the conflict is to override the subproject's junction with a local
+in your project.
+
+You can override junctions in a subproject in the junction declaration
+of that subproject, e.g.:
+
+.. code:: yaml
+
+   kind: junction
+
+   # Here we are junctioning "subproject" which
+   # also junctions "subsubproject", which we also
+   # use directly.
+   #
+   sources:
+   - kind: git
+     url: https://example.com/subproject.git
+
+   config:
+     # Override `subsubproject.bst` in the subproject using
+     # the locally declared `local-subsubproject.bst` junction.
+     #
+     overrides:
+       subsubproject.bst: local-subsubproject.bst
+
+When declaring the ``overrides`` dictionary, the keys (on the left side)
+refer to :ref:`junction paths <format_element_names>` which are relative
+to the subproject you are declaring. The values (on the right side) refer
+to :ref:`junction paths <format_element_names>` which are relative to the
+project in which your junction is declared.
+
+.. warning::
+
+   This approach modifies your subproject, causing its output artifacts
+   to differ from that project's expectations.
+
+   If you rely on validation and guarantees provided by the organization
+   which maintains the subproject, then it is desirable to avoid overriding
+   any details from that upstream project.
+
 
 Linking to other junctions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
-When working with nested junctions, you often need to ensure that multiple
-projects are using the same version of a given subproject.
+Another way to resolve the conflict when your project and a subproject both
+junction a common project, is to simply reuse the same junction from the
+subproject in your toplevel project.
+
+This is preferable to *overrides* because you can avoid modifying the
+subproject you would otherwise be changing with an override.
+
+A convenient way to reuse a nested junction in a higher level project
+is to create a :mod:`link <elements.link>` element to that subproject's
+junction. This will help you avoid redundantly typing out longer
+:ref:`element paths <format_element_names>` in your project's
+:ref:`dependency declarations <format_dependencies>`.
 
-In order to ensure that your project is using a junction to a sub-subproject
-declared by a direct subproject, then you can use a :mod:`link <elements.link>`
-element in place of declaring a junction.
+This way you can simply create the :mod:`link <elements.link>` once
+in your project and use it locally to depend on elements in a nested
+subproject.
 
-This lets you create a link to a junction in the subproject, which you
-can then treat as a regular junction in your toplevel project.
+**Example:**
 
 .. code:: yaml
 
+   # Declare the `subsubproject-link.bst` link element, which
+   # is a symbolic link to the junction declared in the subproject
+   #
    kind: link
 
    config:
      target: subproject.bst:subsubproject.bst
+
+
+.. code:: yaml
+
+   # Depend on elements in the subsubproject using
+   # the subproject's junction directly
+   #
+   kind: autotools
+
+   depends:
+   - subsubproject-link.bst:glibc.bst
+
+
+.. tip::
+
+   When reconciling conflicting junction declarations to the
+   same subproject, it is also possible to use a locally defined
+   :mod:`link <elements.link>` element from one subproject to
+   override another junction to the same project in an adjacent
+   subproject.
+
+
+Multiple project instances
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+By default, loading the same project more than once will result
+in a *conflicting junction error*. There are some use cases which
+demand that you load the same project more than once in the same
+build pipeline.
+
+In order to allow the loading of multiple instances of the same project
+in the same build pipeline, please refer to the
+:ref:`relevant project.conf documentation <project_junctions>`.
 """
 
 from buildstream import Element, ElementError