Restructuring documentation and documenting project optionsoptions-refactor

9 files changed, 693 insertions, 318 deletions

diff --git a/doc/source/completion.rst b/doc/source/completion.rst
deleted file mode 100644
index cc5f6d33c..000000000
--- a/doc/source/completion.rst
+++ /dev/null
@@ -1,16 +0,0 @@
-.. _completion:
-
-Bash Completions
-================
-
-Bash completions are supported by sourcing the ``buildstream/data/bst``
-script found in the BuildStream repository. On many systems this script
-can be installed into a completions directory but when installing BuildStream
-without a package manager this is not an option.
-
-To enable completions for an installation of BuildStream you
-installed yourself from git, just append the script verbatim
-to your ``~/.bash_completion``:
-
-.. literalinclude:: ../../buildstream/data/bst
-   :language: yaml
diff --git a/doc/source/docker.rst b/doc/source/docker.rst
new file mode 100644
index 000000000..74dde77c6
--- /dev/null
+++ b/doc/source/docker.rst
@@ -0,0 +1,31 @@
+.. _docker:
+
+BuildStream inside Docker
+=========================
+The BuildStream project provides
+`Docker images <https://hub.docker.com/r/buildstream/buildstream-fedora/>`_
+containing BuildStream and its dependencies.
+This gives you an easy way to get started using BuildStream on any Unix-like
+platform where Docker is available, including Mac OS X.
+
+To use BuildStream you will need to spawn a container from that image
+and mount your workspace directory as a volume. You will want a second volume
+to store the cache, which we can create from empty like this:
+
+::
+
+    docker volume create buildstream-cache
+
+You can now run the following command to fetch the latest official Docker image
+build, and spawn a container running an interactive shell. This assumes that the
+path to all the source code you need is available in ``~/src``.
+
+::
+
+    docker run -it \
+          --cap-add SYS_ADMIN \
+          --device /dev/fuse \
+          --security-opt seccomp=unconfined \
+          --volume ~/src:/src \
+          --volume buildstream-cache:/root/.cache \
+          buildstream/buildstream-fedora:latest /bin/bash
diff --git a/doc/source/format.rst b/doc/source/format.rst
index 45b3eb47a..7e4638e1c 100644
--- a/doc/source/format.rst
+++ b/doc/source/format.rst
@@ -1,96 +1,10 @@
 .. _format:
 
-The BuildStream Format
-======================
-At the core of BuildStream is a data model of :mod:`Elements <buildstream.element>` which
-are parsed from ``.bst`` files in a project directory and configured from a few different
-sources.
+Element Constructs
+==================
 
-This page should tell you everything you need to know about the base YAML format
-which BuildStream uses.
-
-This will not cover the configurations needed for various plugins, plugin configurations
-are documented in the plugins themselves.
-
-
-The Project Directory
----------------------
-A BuildStream project is a directory consisting of:
-
-* A project configuration file
-* BuildStream element files
-* User defined Plugins
-
-A typical project structure may look like this::
-
-  myproject/project.conf
-  myproject/elements/element1.bst
-  myproject/elements/element2.bst
-  myproject/elements/...
-  myproject/plugins/customelement.py
-  myproject/plugins/customelement.yaml
-  myproject/plugins/...
-
-
-Except for the project configuration file, the user is allowed to structure
-their project directory in any way. For documentation on the format of the project
-configuration file, refer to the :ref:`projectconf` documentation.
-
-Simpler projects may choose to place all element definition files at the
-root of the project directory while more complex projects may decide to
-put stacks in one directory and other floating elements into other directories,
-perhaps placing deployment elements in another directory, this is all fine.
-
-The important part to remember is that when you declare dependency relationships,
-a project relative path to the element one depends on must be provided.
-
-
-Element Composition
--------------------
-Below are the various sources of configuration which go into an element in the order
-in which they are applied. Configurations which are applied later have a higher priority
-and override configurations which precede them.
-
-
-1. Builtin Defaults
-~~~~~~~~~~~~~~~~~~~
-The :ref:`projectconf` provides a set of default values for *variables*
-and the *environment* which are all documented with your copy of BuildStream. 
-
-
-2. Project Configuration
-~~~~~~~~~~~~~~~~~~~~~~~~
-The project wide defaults are now applied on top of builtin defaults. If you specify
-anything in the *variables* or *environment* sections in your ``project.conf`` then it
-will override the builtin defaults.
-
-
-3. Element Defaults
-~~~~~~~~~~~~~~~~~~~
-Elements are all implemented as plugins. Each plugin installs a ``.yaml`` file along side
-their plugin to define the default *variables*, *environment* and *config*. The *config*
-is element specific and as such this is the first place where defaults can be set on the
-*config* section.
-
-The *variables* and *environment* specified in the declaring plugin's defaults here override
-the project configuration defaults for the given element ``kind``.
-
-
-4. Project Configuration
-~~~~~~~~~~~~~~~~~~~~~~~~
-The ``project.conf`` now gives you another opportunity to override *variables*, *environment*
-and *config* sections on a per element basis.
-
-Configurations specified in the *elements* section of the ``project.conf`` will override
-the given element's default.
-
-
-5. Element Declarations
-~~~~~~~~~~~~~~~~~~~~~~~
-Finally, after having resolved any `Architecture Conditionals`_
-in the parsing phase of loading element declarations; the configurations specified in a
-``.bst`` file have the last word on any configuration in the data model.
 
+.. _format_basics:
 
 Element Basics
 --------------
@@ -409,8 +323,8 @@ dependency and that all referenced variables are declared, the following is fine
 
 .. note::
 
-   It should be noted that variable resolution only happens after all `Element Composition`_
-   has already taken place.
+   It should be noted that variable resolution only happens after all
+   :ref:`Element Composition <format_composition>` has already taken place.
 
    This is to say that overriding ``%{version}`` at a higher priority will effect
    the final result of ``%{release-text}``.
@@ -432,61 +346,3 @@ dependency and that all referenced variables are declared, the following is fine
      install-commands:
      - |
        %{make-install} RELEASE_TEXT="%{release-text}"
-
-
-Architecture Conditionals
--------------------------
-To BuildStream, an architecture is simply an arbitrary name that is associated with
-the target architecture and compiler tuning. Conditional YAML segments can be applied
-for a given target architecture, like so:
-
-.. code:: yaml
-
-   kind: autotools
-   config:
-     something: 5
-   arches:
-     x86_64:
-       config:
-         something: 6
-     x86_32:
-       config:
-         something: 7
-
-The ``arches`` attribute, if provided, overrides the element for a given architecture
-name. It is not considered an error if the element does not provide an architecture
-clause for the specific architecture BuildStream was launched to build for.
-
-There is also a ``host-arches`` attribute, which operates in the same manner but
-follows the *host* architecture rather than the *target* architecture.
-
-In the above example we demonstrate that a given ``config`` attribute can be overridden
-by an architecture conditional, this can however be done for any segment of the
-element such as ``depends``, ``sources`` and ``public`` as well. It is however illegal
-to override the element ``kind`` in any conditional.
-
-Further, it should be noted that when applying elements to a list in the element YAML,
-the conditional segments are *appended* to the parent list and do not replace the list
-entirely.
-
-Consider for example:
-
-.. code:: yaml
-
-   kind: autotools
-   depends:
-   - elements/foo.bst
-   arches:
-     x86_64:
-       depends:
-       - elements/bar.bst
-
-When targetting the ``x86_64`` architecture name, the above element YAML will
-expand to the following YAML:
-
-.. code:: yaml
-
-   kind: autotools
-   depends:
-   - elements/foo.bst
-   - elements/bar.bst
diff --git a/doc/source/formatintro.rst b/doc/source/formatintro.rst
new file mode 100644
index 000000000..42e7d3804
--- /dev/null
+++ b/doc/source/formatintro.rst
@@ -0,0 +1,171 @@
+.. _formatintro:
+
+
+Introduction
+============
+At the core of BuildStream is a data model of :mod:`Elements <buildstream.element>` which
+are parsed from ``.bst`` files in a project directory and configured from a few different
+sources.
+
+When BuildStream loads your project, various levels of composition occur, allowing
+configuration on various levels with different priority.
+
+This page provides an introduction to the project directory structure, explains the
+basic *directives* supported inherently throughout the format, and outlines how composition
+occurs and what configurations are considered in which order.
+
+The meaning of the various constructs expressed in the BuildStream format are covered
+in other sections of the documentation.
+
+.. _format_structure:
+
+
+Directory Structure
+-------------------
+A BuildStream project is a directory consisting of:
+
+* A project configuration file
+* BuildStream element files
+* User defined Plugins
+
+A typical project structure may look like this::
+
+  myproject/project.conf
+  myproject/elements/element1.bst
+  myproject/elements/element2.bst
+  myproject/elements/...
+  myproject/plugins/customelement.py
+  myproject/plugins/customelement.yaml
+  myproject/plugins/...
+
+
+Except for the project configuration file, the user is allowed to structure
+their project directory in any way. For documentation on the format of the project
+configuration file, refer to the :ref:`projectconf` documentation.
+
+Simpler projects may choose to place all element definition files at the
+root of the project directory while more complex projects may decide to
+put stacks in one directory and other floating elements into other directories,
+perhaps placing deployment elements in another directory, this is all fine.
+
+The important part to remember is that when you declare dependency relationships,
+a project relative path to the element one depends on must be provided.
+
+
+.. _format_directives:
+
+Directives
+----------
+
+(?) Conditionals
+~~~~~~~~~~~~~~~~
+The ``(?)`` directive allows expression of conditional statements which
+test :ref:`project option <project_options>` values.
+
+The ``(?)`` directive may appear as a key in any dictionary expressed
+in YAML, and it's value is a list of conditional expressions. Each conditional
+expression must be a single key dictionary, where the key is the conditional
+expression itself, and the value is a dictionary to be composited into the
+parent dictionary containing the ``(?)`` directive if the expression evaluates
+to a truthy value.
+
+**Example:**
+
+.. code:: yaml
+
+   variables:
+     prefix: "/usr"
+     enable-debug: False
+     (?):
+     - relocate == True:
+         prefix: "/opt"
+     - debug == True:
+         enable-debug: True
+
+
+Expressions are evaluated in the specified order, and each time an expression
+evaluates to a truthy value, it's value will be composited to the parent dictionary
+in advance of processing other elements, allowing for logically overriding previous
+decisions in the condition list.
+
+Nesting of conditional statements is also supported.
+
+**Example:**
+
+.. code:: yaml
+
+   variables:
+     enable-logging: False
+     enable-debug: False
+     (?):
+     - logging == True:
+         enable-logging: True
+         (?):
+	 - debugging == True:
+             enable-debug: True
+
+
+Conditionals are expressed in a pythonic syntax, the specifics for
+testing the individually supported option types are described in
+their :ref:`respective documentation <project_options>`.
+
+Compound conditionals are also allowed.
+
+**Example:**
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - (logging == True and debugging == True):
+         enable-debug: True
+
+
+.. _format_composition:
+
+Composition
+-----------
+Below are the various sources of configuration which go into an element in the order
+in which they are applied. Configurations which are applied later have a higher priority
+and override configurations which precede them.
+
+
+1. Builtin Defaults
+~~~~~~~~~~~~~~~~~~~
+The :ref:`projectconf` provides a set of default values for *variables*
+and the *environment* which are all documented with your copy of BuildStream. 
+
+
+2. Project Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~
+The project wide defaults are now applied on top of builtin defaults. If you specify
+anything in the *variables* or *environment* sections in your ``project.conf`` then it
+will override the builtin defaults.
+
+
+3. Element Defaults
+~~~~~~~~~~~~~~~~~~~
+Elements are all implemented as plugins. Each plugin installs a ``.yaml`` file along side
+their plugin to define the default *variables*, *environment* and *config*. The *config*
+is element specific and as such this is the first place where defaults can be set on the
+*config* section.
+
+The *variables* and *environment* specified in the declaring plugin's defaults here override
+the project configuration defaults for the given element ``kind``.
+
+
+4. Project Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~
+The ``project.conf`` now gives you another opportunity to override *variables*, *environment*
+and *config* sections on a per element basis.
+
+Configurations specified in the *elements* section of the ``project.conf`` will override
+the given element's default.
+
+
+5. Element Declarations
+~~~~~~~~~~~~~~~~~~~~~~~
+Finally, after having resolved any `Architecture Conditionals`_
+in the parsing phase of loading element declarations; the configurations specified in a
+``.bst`` file have the last word on any configuration in the data model.
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 70c7354fe..cbb320438 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -18,84 +18,57 @@ These pipelines are composed of abstract elements which perform mutations on
 on *filesystem data* as input and output, and are related to eachother by their
 dependencies.
 
-Basic element types for importing SDKs in the form of tarballs or ostree checkouts,
-building software components and exporting SDKs or deploying bootable filesystem images
-will be included in BuildStream, but it is expected that projects forge their own custom
-elements for doing more elaborate things such as deploying software in special ways.
 
+Installing
+----------
+* :ref:`installing`
+* :ref:`docker`
+* :ref:`artifacts`
 
-Using BuildStream
-=================
-Here are some resources to help understand the BuildStream format and also to get
-off the ground installing it on a new host.
 
-* :ref:`installing`
+Running
+-------
 * :ref:`invoking`
-* :ref:`completion`
 * :ref:`config`
-* :ref:`projectconf`
-* :ref:`format`
-* :ref:`public`
-* :ref:`artifacts`
 
 
-Elements
---------
+Project format
+--------------
+* :ref:`formatintro`
 
+  * :ref:`format_structure`
+  * :ref:`format_directives`
+  * :ref:`format_composition`
 
-General Elements
-~~~~~~~~~~~~~~~~
+* :ref:`projectconf`
 
-* :mod:`stack <elements.stack>` - Symbolic Element for dependency grouping
-* :mod:`import <elements.import>` - Import sources directly
-* :mod:`compose <elements.compose>` - Compose the output of multiple elements
-* :mod:`script <elements.script>` - Run scripts to create output
-* :mod:`x86image <elements.x86image>` - Create x86 bootable disk images
-* :mod:`dpkg_deploy <elements.dpkg_deploy>` - Create debian packages
+  * :ref:`project_essentials`
+  * :ref:`project_options`
+  * :ref:`project_defaults`
+  * :ref:`project_builtin_defaults`
 
-Build Elements
-~~~~~~~~~~~~~~
+* :ref:`format`
 
-* :mod:`manual <elements.manual>` - Manual Build Element
-* :mod:`autotools <elements.autotools>` - Autotools Build Element
-* :mod:`cmake <elements.cmake>` - CMake Build Element
-* :mod:`qmake <elements.qmake>` - QMake Build Element
-* :mod:`distutils <elements.distutils>` - Python Distutils Build Element
-* :mod:`makemaker <elements.makemaker>` - Perl MakeMaker Build Element
-* :mod:`modulebuild <elements.modulebuild>` - Perl Module::Build Build Element
-* :mod:`meson <elements.meson>` - Meson Build Element
-* :mod:`dpkg_build <elements.dpkg_build>` - Dpkg-based build element
-* :mod:`pip <elements.pip>` - Pip build element
+  * :ref:`format_basics`
+  * :ref:`format_dependencies`
+  * :ref:`format_variables`
 
+* :ref:`public`
 
-Sources
---------
-The following source types are provided with BuildStream:
 
-* :mod:`local <sources.local>` - A Source implementation for local files and directories
-* :mod:`tar <sources.tar>` - A Source implementation for tarballs
-* :mod:`git <sources.git>` - A Source implementation for git
-* :mod:`bzr <sources.bzr>` - A Source implementation for bazaar
-* :mod:`ostree <sources.ostree>` - A Source implementation for ostree
-* :mod:`patch <sources.patch>` - A Source implementation for applying local patches
+Builtin Plugins
+---------------
+* :ref:`plugins`
 
-.. _core_framework:
+  * :ref:`plugins_elements`
+  * :ref:`plugins_sources`
 
-Core Framework
---------------
-The core public APIs are of interest to anyone who wishes to
-implement custom :mod:`Element <buildstream.element>` or
-:mod:`Source <buildstream.source>` plugins.
-
-* :mod:`Plugin <buildstream.plugin>` - Base Class for all plugins
-* :mod:`Source <buildstream.source>` - Base Source Class
-* :mod:`Element <buildstream.element>` - Base Element Class
-* :mod:`BuildElement <buildstream.buildelement>` - Build Element Class
-* :mod:`ScriptElement <buildstream.scriptelement>` - Script Element Class
-* :mod:`Context <buildstream.context>` - Invocation Context
-* :mod:`Project <buildstream.project>` - Loaded Project
-* :mod:`Sandbox <buildstream.sandbox.sandbox>` - Build Sandbox
-* :mod:`Utilities <buildstream.utils>` - Utilities for Plugins
+
+Creating Plugins
+----------------
+* :ref:`pluginauthoring`
+
+  * :ref:`core_framework`
 
 
 Indices and tables
diff --git a/doc/source/install.rst b/doc/source/install.rst
index 6952bec44..096702f64 100644
--- a/doc/source/install.rst
+++ b/doc/source/install.rst
@@ -1,8 +1,8 @@
 .. _installing:
 
 
-Installing BuildStream
-======================
+BuildStream on your host
+========================
 Until BuildStream is available in your distro, there are a few hoops to jump
 through to get started.
 
@@ -11,8 +11,8 @@ then we have some instructions below which can get you started using BuildStream
 within a Docker container.
 
 
-Installing base system requirements
------------------------------------
+System requirements
+-------------------
 BuildStream requires the following base system requirements:
 
 * python3 >= 3.4
@@ -90,6 +90,21 @@ A regular way to do this is to add the following line to the end of your ``~/.ba
   export PATH=${PATH}:~/.local/bin
 
 
+Bash Completions
+~~~~~~~~~~~~~~~~
+Bash completions are supported by sourcing the ``buildstream/data/bst``
+script found in the BuildStream repository. On many systems this script
+can be installed into a completions directory but when installing BuildStream
+without a package manager this is not an option.
+
+To enable completions for an installation of BuildStream you
+installed yourself from git, just append the script verbatim
+to your ``~/.bash_completion``:
+
+.. literalinclude:: ../../buildstream/data/bst
+   :language: yaml
+
+
 Upgrading with pip
 ~~~~~~~~~~~~~~~~~~
 To upgrade a previously install BuildStream, you will need to pull the latest
@@ -99,86 +114,3 @@ changes and reinstall as such::
   cd buildstream
   git pull --rebase
   pip3 install --user .
-
-
-Using virtualenv
-----------------
-If you want to install BuildStream in such a way that ``pip`` does not add
-any files to your home directory, you can use virtualenv. This is a bit less
-convenient because it requires you enter a special environment every time you
-want to use BuildStream.
-
-To use virtualenv, you will first need to install virtualenv with your
-package manager, in addition to the base requirements listed above.
-
-E.g. with debian systems::
-
-  sudo apt-get install python3-virtualenv
-
-At this point the following instructions will get you a virtual python
-environment that is completely encapsulated and does not interfere with
-your system or home directory::
-
-  # Clone the repository
-  git clone https://gitlab.com/BuildStream/buildstream.git
-  cd buildstream
-
-  # Create a virtualenv sandbox for the installation, you need to
-  # enable the system site packages in order to have access to the
-  # ostree python bindings which unfortunately cannot be installed
-  # with pip into your sandbox
-  virtualenv --system-site-packages -p python3 sandbox
-
-  # Install into the virtualenv using pip inside the virtualenv
-  ./sandbox/bin/pip3 install .
-
-Once you have done the above, you have a completely disposable
-``sandbox`` directory which provides an environment you can enter
-at anytime to use BuildStream. BuildStream man pages should also
-be available when in the virtualenv environment.
-
-To enter the environment, source it's activation script::
-
-  source sandbox/bin/activate
-
-From here, the ``bst`` command is available, run ``bst --help`` or ``man bst``.
-
-The activation script adds a bash function to your environment which you
-can use to exit the sandbox environment, just type ``deactivate`` in the
-shell to deactivate the virtualenv sandbox.
-
-To upgrade to a new version of BuildStream when using virtualenv, just
-remove the ``sandbox`` directory completely and recreate it with a new
-version of BuildStream.
-
-
-Using BuildStream inside Docker
-===============================
-
-The BuildStream project provides
-`Docker images <https://hub.docker.com/r/buildstream/buildstream-fedora/>`_
-containing BuildStream and its dependencies.
-This gives you an easy way to get started using BuildStream on any Unix-like
-platform where Docker is available, including Mac OS X.
-
-To use BuildStream build tool you will need to spawn a container from that image
-and mount your workspace directory as a volume. You will want a second volume
-to store the cache, which we can create from empty like this:
-
-::
-
-    docker volume create buildstream-cache
-
-You can now run the following command to fetch the latest official Docker image
-build, and spawn a container running an interactive shell. This assumes that the
-path to all the source code you need is available in ``~/src``.
-
-::
-
-    docker run -it \
-          --cap-add SYS_ADMIN \
-          --device /dev/fuse \
-          --security-opt seccomp=unconfined \
-          --volume ~/src:/src \
-          --volume buildstream-cache:/root/.cache \
-          buildstream/buildstream-fedora:latest /bin/bash
diff --git a/doc/source/pluginindex.rst b/doc/source/pluginindex.rst
new file mode 100644
index 000000000..b50dc8a22
--- /dev/null
+++ b/doc/source/pluginindex.rst
@@ -0,0 +1,51 @@
+.. _plugins:
+
+Plugins
+=======
+
+
+.. _plugins_elements:
+
+Elements
+--------
+The following element types are provided with BuildStream:
+
+
+General Elements
+~~~~~~~~~~~~~~~~
+
+* :mod:`stack <elements.stack>` - Symbolic Element for dependency grouping
+* :mod:`import <elements.import>` - Import sources directly
+* :mod:`compose <elements.compose>` - Compose the output of multiple elements
+* :mod:`script <elements.script>` - Run scripts to create output
+* :mod:`x86image <elements.x86image>` - Create x86 bootable disk images
+* :mod:`dpkg_deploy <elements.dpkg_deploy>` - Create debian packages
+
+Build Elements
+~~~~~~~~~~~~~~
+
+* :mod:`manual <elements.manual>` - Manual Build Element
+* :mod:`autotools <elements.autotools>` - Autotools Build Element
+* :mod:`cmake <elements.cmake>` - CMake Build Element
+* :mod:`qmake <elements.qmake>` - QMake Build Element
+* :mod:`distutils <elements.distutils>` - Python Distutils Build Element
+* :mod:`makemaker <elements.makemaker>` - Perl MakeMaker Build Element
+* :mod:`modulebuild <elements.modulebuild>` - Perl Module::Build Build Element
+* :mod:`meson <elements.meson>` - Meson Build Element
+* :mod:`dpkg_build <elements.dpkg_build>` - Dpkg-based build element
+* :mod:`pip <elements.pip>` - Pip build element
+
+
+.. _plugins_sources:
+
+
+Sources
+--------
+The following source types are provided with BuildStream:
+
+* :mod:`local <sources.local>` - A Source implementation for local files and directories
+* :mod:`tar <sources.tar>` - A Source implementation for tarballs
+* :mod:`git <sources.git>` - A Source implementation for git
+* :mod:`bzr <sources.bzr>` - A Source implementation for bazaar
+* :mod:`ostree <sources.ostree>` - A Source implementation for ostree
+* :mod:`patch <sources.patch>` - A Source implementation for applying local patches
diff --git a/doc/source/projectconf.rst b/doc/source/projectconf.rst
index 54c5e80db..2f4d997bc 100644
--- a/doc/source/projectconf.rst
+++ b/doc/source/projectconf.rst
@@ -9,9 +9,382 @@ aliases relevant for the sources used in the given project as well as
 overrides for the configuration of element types used in the project.
 
 Values specified in the project configuration override any of the
-default BuildStream project configuration, which is included here
-for reference and includes comments describing all of the possible
-configuration values:
+default BuildStream project configuration, which is included
+:ref:`here <project_defaults>` for reference.
+
+
+.. _project_essentials:
+
+Essentials
+----------
+
+
+Project Name
+~~~~~~~~~~~~
+The first thing to setup in your ``project.conf`` should be the name
+of your project.
+
+.. code:: yaml
+
+   name: my-project-name
+
+The project name will be used in user configuration and anywhere
+that a project needs to be specified.
+
+
+Element Path
+~~~~~~~~~~~~
+To allow the user to structure their project nicely, BuildStream
+allows the user to specify a project subdirectory where element
+``.bst`` files are stored.
+
+.. code:: yaml
+
+   element-path: elements
+
+Note that elements are referred to by their relative paths, whenever
+elements are referred to in a ``.bst`` file or on the command line.
+
+
+Source Aliases
+~~~~~~~~~~~~~~
+In order to abstract the download location of source code and
+any assets which need to be downloaded, and also as a matter of
+convenience, BuildStream allows one to create named aliases for
+URLs which are to be used in the individual ``.bst`` files.
+
+.. code:: yaml
+
+   aliases:
+     foo: git://git.foo.org/
+     bar: http://bar.com/downloads/
+
+
+Artifact Server
+~~~~~~~~~~~~~~~
+If you have setup an :ref:`artifact server <artifacts>` for your
+project then it is convenient to configure this in your ``project.conf``
+so that users need not have any additional configuration to communicate
+with an artifact share.
+
+.. code:: yaml
+
+  artifacts:
+
+    # A url from which to download prebuilt artifacts
+    pull-url: https://foo.com/artifacts
+
+    # A url to upload built artifacts to
+    # (must point to the same repository as pull-url)
+    push-url: artifacts@foo.com:artifacts
+
+    # Specify the port number for pushing artifacts, if it's
+    # not the default port 22
+    push-port: 10000
+
+
+Plugin Paths
+~~~~~~~~~~~~
+If your project includes any custom *Elements* or *Sources*, then
+the project relative subdirectory where these plugins are stored
+must be specified.
+
+.. code:: yaml
+
+   plugins:
+
+     elements:
+     - plugins/local-elements
+     - plugins/shared-elements
+
+     sources:
+     - plugins/local-sources
+
+
+Versioning
+~~~~~~~~~~
+The BuildStream format is guaranteed to be backwards compatible
+with any earlier releases. The core YAML format, the format supported
+by various plugins, and the overall BuildStream release version are
+revisioned separately.
+
+The ``project.conf`` allows asserting the minimal required core
+format version and the minimal required version for individual
+plugins.
+
+.. code:: yaml
+
+  required-versions:
+
+    # The minimum base BuildStream format
+    project: 0
+
+    # The minimum version of the autotools element
+    elements:
+      autotools: 3
+
+
+
+.. _project_options:
+
+Options
+-------
+Options are how BuildStream projects can define parameters which
+can be configured by users invoking BuildStream to build your project.
+
+Options are declared in the ``project.conf`` in the main ``options``
+dictionary.
+
+.. code:: yaml
+
+   options:
+     debug:
+       type: bool
+       description: Whether to enable debugging
+       default: False
+
+
+Boolean
+~~~~~~~
+The ``bool`` option type allows specifying boolean values which
+can be cased in conditional expressions.
+
+.. code:: yaml
+
+   options:
+     debug:
+       type: bool
+       description: Whether to enable debugging
+       default: False
+
+Boolean options can be tested in expressions with equality tests:
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - debug == True:
+         enable-debug: True
+
+Or simply treated as truthy values:
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - debug:
+         enable-debug: True
+
+Enumeration
+~~~~~~~~~~~
+The ``enum`` option type allows specifying a string value
+with a restricted set of possible values.
+
+.. code:: yaml
+
+   options:
+     loglevel:
+       type: enum
+       description: The logging level
+       values:
+       - debug
+       - info
+       - warning
+       default: info
+
+Enumeration options must be tested as strings in conditional
+expressions:
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - loglevel == "debug":
+         enable-debug: True
+
+
+Flags
+~~~~~
+The ``flags`` option type allows specifying a list of string
+values with a restricted set of possible values.
+
+In contrast with the ``enum`` option type, the *default* value
+need not be specified and will default to an empty set.
+
+.. code:: yaml
+
+   options:
+     logmask:
+       type: flags
+       description: The logging mask
+       values:
+       - debug
+       - info
+       - warning
+       default:
+       - info
+
+Flags type options can be tested in conditional expressions using
+a pythonic *in* syntax to test if an element is present in a set:
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - ("debug" in logmask):
+         enable-debug: True
+
+
+Architecture
+~~~~~~~~~~~~
+The ``arch`` type option is special enumeration option which
+defaults to the result of `uname -m`, and does not support
+assigning any default in the project configuration.
+
+.. code:: yaml
+
+   options:
+     machine_arch:
+       type: arch
+       description: The machine architecture
+       values:
+       - arm
+       - aarch64
+       - i386
+       - x86_64
+
+
+Architecture options can be tested with the same expressions
+as other Enumeration options.
+
+
+Element Mask
+~~~~~~~~~~~~
+The ``element-mask`` option type is a special Flags option
+which automatically allows only element names as values.
+
+.. code:: yaml
+
+   options:
+     debug_elements:
+       type: element-mask
+       description: The elements to build in debug mode
+
+This can be convenient for automatically declaring an option
+which might apply to any element, and can be tested with the
+same syntax as other Flag options.
+
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - ("element.bst" in debug_elements):
+         enable-debug: True
+
+
+
+.. _project_defaults:
+
+Specifying Defaults
+--------------------
+The ``project.conf`` plays a role in defining elements by
+providing default values and also by overriding values declared
+by plugins on a plugin wide basis.
+
+See the :ref:`composition <format_composition>` documentation for
+more detail on how elements are composed.
+
+
+Variables
+~~~~~~~~~
+The defaults for :ref:`Variables <format_variables>` used in your
+project is defined here.
+
+.. code:: yaml
+
+   variables:
+     prefix: "/usr"
+
+
+Environment
+~~~~~~~~~~~
+The defaults environment for the build sandbox is defined here.
+
+.. code:: yaml
+
+   environment:
+     PATH: /usr/bin:/bin:/usr/sbin:/sbin
+
+Additionally, the special ``environment-nocache`` list which specifies
+which environment variables do not effect build output, and are thus
+not considered in the calculation of artifact keys can be defined here.
+
+.. code:: yaml
+
+   environment-nocache:
+   - MAXJOBS
+
+Note that the ``environment-nocache`` list only exists so that we can
+control parameters such as ``make -j ${MAXJOBS}``, allowing us to control
+the number of jobs for a given build without effecting the resulting
+cache key.
+
+
+Split Rules
+~~~~~~~~~~~
+The project wide :ref:`split rules <public_split_rules>` defaults can
+be specified here.
+
+.. code:: yaml
+
+   split-rules:
+     devel:
+     - |
+       %{includedir}
+     - |
+       %{includedir}/**
+     - |
+       %{libdir}/lib*.a
+     - |
+       %{libdir}/lib*.la
+
+
+Element Overrides
+~~~~~~~~~~~~~~~~~
+Base attributes declared by element default yaml files can be overridden
+on a project wide basis. The elements dictionary can be used to override
+variables, environments or plugin specific configuration data as shown below.
+
+
+.. code:: yaml
+
+   elements:
+
+     # Override default values for all autotools elements
+     autotools:
+
+       variables:
+         bindir: "%{prefix}/bin"
+
+       config:
+         configure-commands: ...
+
+       environment:
+         PKG_CONFIG_PATH=%{libdir}/pkgconfig
+
+
+.. _project_builtin_defaults:
+
+Builtin Defaults
+----------------
+BuildStream defines some default values for convenience, the default
+values overridden by your project's ``project.conf`` are presented here:
 
   .. literalinclude:: ../../buildstream/data/projectconfig.yaml
      :language: yaml
diff --git a/doc/source/public.rst b/doc/source/public.rst
index 298fc6158..eb7381ec6 100644
--- a/doc/source/public.rst
+++ b/doc/source/public.rst
@@ -1,8 +1,8 @@
 .. _public:
 
 
-Public Data
-===========
+Builtin Public Data
+===================
 
 Elements can provide public data which can be read by other elements
 later in the pipeline, the format for exposing public data on a given
@@ -15,6 +15,8 @@ completely in the ``bst`` domain.
 In this section we will describe the public data in the ``bst`` domain.
 
 
+.. _public_integration:
+
 Integration Commands
 --------------------
 
@@ -42,6 +44,8 @@ will always integrate the build dependencies after staging and before running
 any build commands.
 
 
+.. _public_split_rules:
+
 Split Rules
 -----------