Documenting project.refs

  o Adding explanation of the `ref-storage` option in project.conf

  o Adjusting some adjacent relevant documentation

  o Adding a new projectrefs.rst section for the project.refs
    format itself, and linking it from the main authoring page.

4 files changed, 143 insertions, 21 deletions

diff --git a/doc/source/formatintro.rst b/doc/source/formatintro.rst
index a1f4e8f26..f3e7a2874 100644
--- a/doc/source/formatintro.rst
+++ b/doc/source/formatintro.rst
@@ -28,11 +28,13 @@ A BuildStream project is a directory consisting of:
 
 * A project configuration file
 * BuildStream element files
-* User defined Plugins
+* Optional user defined plugins
+* An optional project.refs file
 
 A typical project structure may look like this::
 
   myproject/project.conf
+  myproject/project.refs
   myproject/elements/element1.bst
   myproject/elements/element2.bst
   myproject/elements/...
diff --git a/doc/source/main_authoring.rst b/doc/source/main_authoring.rst
index f23bfb7c1..14d051276 100644
--- a/doc/source/main_authoring.rst
+++ b/doc/source/main_authoring.rst
@@ -34,14 +34,23 @@ Project format
 
 * :ref:`public`
 
+  * :ref:`public_integration`
+  * :ref:`public_split_rules`
+  * :ref:`public_overlap_whitelist`
+
+* :ref:`projectrefs`
+
+  * :ref:`projectrefs_basics`
+  * :ref:`projectrefs_format`
+
 
 Plugins
 -------
 Plugins provide their own individual plugin specific YAML configurations,
 The element ``.bst`` files can specify plugin specific configuration in
 the :ref:`config section <format_config>`, while sources declared on a
-given element specify their plugin specific configuration
-:ref:`directly on their main dictionary <format_sources>`.
+given element specify their plugin specific configuration directly
+:ref:`in their source declarations <format_sources>`.
 
 
 Elements
diff --git a/doc/source/projectconf.rst b/doc/source/projectconf.rst
index 0db06fad8..248f140ab 100644
--- a/doc/source/projectconf.rst
+++ b/doc/source/projectconf.rst
@@ -21,6 +21,8 @@ Essentials
 ----------
 
 
+.. _project_format_name:
+
 Project Name
 ~~~~~~~~~~~~
 The first thing to setup in your ``project.conf`` should be the name
@@ -74,6 +76,60 @@ Note that elements are referred to by their relative paths, whenever
 elements are referred to in a ``.bst`` file or on the command line.
 
 
+.. _project_format_ref_storage:
+
+Ref Storage
+~~~~~~~~~~~
+By default, BuildStream expects to read and write source references
+directly in the :ref:`source declaration <format_sources>`, but this
+can be inconvenient and prohibitive in some workflows.
+
+Alternatively, BuildStream allows source references to be stored
+centrally in a :ref:`project.refs file <projectrefs>` in the toplevel
+:ref:`project directory <format_structure>`.
+
+This can be controlled with the ``ref-storage`` option, which is
+allowed to be configured with the following values:
+
+* ``inline``
+
+  Source references are stored directly in the
+  :ref:`source declaration <format_sources>`
+
+* ``project.refs``
+
+  Source references are stored in the ``project.refs`` file.
+
+To enable storing of source references in ``project.refs``, add the
+following to your ``project.conf``:
+
+.. code:: yaml
+
+  ref-storage: project.refs
+
+.. note::
+
+   The ``ref-storage`` configuration is available since :ref:`format version 5 <project_format_version>`
+
+
+Fail on Overlaps
+~~~~~~~~~~~~~~~~
+When multiple elements are staged, there's a possibility that different
+elements will try and stage different versions of the same file.
+
+When ``fail-on-overlap`` is true, if an overlap is detected
+that hasn't been allowed by the element's
+:ref:`overlap whitelist<public_overlap_whitelist>`,
+then an error will be raised and the build will fail.
+
+otherwise, a warning will be raised indicating which files had overlaps,
+and the order that the elements were overlapped.
+
+.. code:: yaml
+
+  fail-on-overlap: true
+
+
 Source Aliases
 ~~~~~~~~~~~~~~
 In order to abstract the download location of source code and
@@ -108,24 +164,6 @@ You can also specify a list of caches here; earlier entries in the list
 will have higher priority than later ones.
 
 
-Fail on Overlaps
-~~~~~~~~~~~~~~~~
-When multiple elements are staged, there's a possibility that different
-elements will try and stage different versions of the same file.
-
-When ``fail-on-overlap`` is true, if an overlap is detected
-that hasn't been allowed by the element's
-:ref:`overlap whitelist<public_overlap_whitelist>`,
-then an error will be raised and the build will fail.
-
-otherwise, a warning will be raised indicating which files had overlaps,
-and the order that the elements were overlapped.
-
-.. code:: yaml
-
-  fail-on-overlap: true
-
-
 .. _project_plugins:
 
 External Plugins
diff --git a/doc/source/projectrefs.rst b/doc/source/projectrefs.rst
new file mode 100644
index 000000000..557f8c5f3
--- /dev/null
+++ b/doc/source/projectrefs.rst
@@ -0,0 +1,73 @@
+:orphan:
+
+
+.. _projectrefs:
+
+The project.refs File
+=====================
+If one has elected to store source references in a single ``project.refs``
+file, then it will be stored at the toplevel of your project directory
+adjacent to ``project.conf``. This can be configured in your project
+using the :ref:`ref-storage configuration <project_format_ref_storage>`
+
+
+.. _projectrefs_basics:
+
+Basic Behavior
+--------------
+When a ``project.refs`` file is in use, any source references found
+in the :ref:`inline source declarations <format_sources>` are considered
+invalid and will be ignored, and a warning will be emitted for them.
+
+When ``bst track`` is run for your project, the ``project.refs`` file
+will be updated instead of the inline source declarations. In the absence
+of a ``project.refs`` file, ``bst track`` will create one automatically
+with the tracking results.
+
+An interesting property of ``project.refs`` is that it allows for
+*cross junction tracking*. This is to say that it is possible to override
+the *ref* of a given source in a project that your project depends on via
+a :mod:`junction <elements.junction>`, without actually modifying the
+junctioned project.
+
+
+.. _projectrefs_format:
+
+Format
+------
+The ``project.refs`` uses the same YAML format used throughout BuildStream,
+and supports the same :ref:`directives <format_directives>` which apply to
+``project.conf`` and element declaration files (i.e. *element.bst* files).
+
+The ``project.refs`` file format itself is very simple, it contains a single ``projects``
+key at the toplevel, which is a dictionary of :ref:`project names <project_format_name>`.
+Each *project name* is a dictionary of *element names*, and each *element name* holds
+a list of dictionaries corresponding to the element's :ref:`sources <format_sources>`.
+
+
+**Example**
+
+.. code:: yaml
+
+   # The local project's name is "core"
+   core:
+
+     # A dictionary of element names
+     base/automake.bst:
+
+     # A list of sources corresponding to the element
+     # in the same order in which they were declared.
+     #
+     # The values of this list are dictionaries of the
+     # symbolic "ref" portion understood by the given
+     # source plugin implementation.
+     # 
+     - ref: af6ba39142220687c500f79b4aa2f181d9b24e4...
+
+   # The "core" project depends on the "bootstrap" project,
+   # here we are allowed to override the refs for the projects
+   # we depend on through junctions.
+   bootstrap:
+
+     zlib.bst:
+     - ref: 4ff941449631ace0d4d203e3483be9dbc9da4540...