Restructure directories under source/ to resemble ToC

1 files changed, 802 insertions, 0 deletions

diff --git a/doc/source/index/authoring/projectconf.rst b/doc/source/index/authoring/projectconf.rst
new file mode 100644
index 000000000..9fe28cea2
--- /dev/null
+++ b/doc/source/index/authoring/projectconf.rst
@@ -0,0 +1,802 @@
+
+.. _projectconf:
+
+
+Project Configuration
+=====================
+The project configuration file should be named ``project.conf`` and
+be located at the project root. It holds information such as Source
+aliases relevant for the sources used in the given project as well as
+overrides for the configuration of element types used in the project.
+
+Values specified in the project configuration override any of the
+default BuildStream project configuration, which is included
+:ref:`here <project_defaults>` for reference.
+
+
+.. _project_essentials:
+
+Essentials
+----------
+
+
+.. _project_format_name:
+
+Project Name
+~~~~~~~~~~~~
+The first thing to setup in your ``project.conf`` should be the name
+of your project.
+
+.. code:: yaml
+
+   name: my-project-name
+
+The project name will be used in user configuration and anywhere
+that a project needs to be specified.
+
+
+.. _project_format_version:
+
+Format Version
+~~~~~~~~~~~~~~
+The BuildStream format is guaranteed to be backwards compatible
+with any earlier releases. The project's minimum required format
+version of BuildStream can be specified in ``project.conf`` with
+the ``format-version`` field, e.g.:
+
+.. code:: yaml
+
+  # The minimum base BuildStream format
+  format-version: 0
+
+BuildStream will increment it's core YAML format version at least once
+in any given minor point release where the format has been extended
+to support a new feature.
+
+.. note::
+
+   :mod:`Element <buildstream.element>` and :mod:`Source <buildstream.source>`
+   plugins also implement their own YAML configuration fragments and as
+   such are revisioned separately from the core format. See :ref:`project_plugins`
+   for details on specifying a minimum version of a specific plugin.
+
+   Core :mod:`Elements <buildstream.element>` and :mod:`Sources <buildstream.source>`
+   which are maintained and distributed as a part of BuildStream are revisioned
+   under the same global ``format-version`` described here.
+
+
+.. _project_element_path:
+
+Element Path
+~~~~~~~~~~~~
+To allow the user to structure their project nicely, BuildStream
+allows the user to specify a project subdirectory where element
+``.bst`` files are stored.
+
+.. code:: yaml
+
+   element-path: elements
+
+Note that elements are referred to by their relative paths, whenever
+elements are referred to in a ``.bst`` file or on the command line.
+
+
+.. _project_format_ref_storage:
+
+Ref Storage
+~~~~~~~~~~~
+By default, BuildStream expects to read and write source references
+directly in the :ref:`source declaration <format_sources>`, but this
+can be inconvenient and prohibitive in some workflows.
+
+Alternatively, BuildStream allows source references to be stored
+centrally in a :ref:`project.refs file <projectrefs>` in the toplevel
+:ref:`project directory <format_structure>`.
+
+This can be controlled with the ``ref-storage`` option, which is
+allowed to be configured with the following values:
+
+* ``inline``
+
+  Source references are stored directly in the
+  :ref:`source declaration <format_sources>`
+
+* ``project.refs``
+
+  Source references are stored in the ``project.refs`` file.
+
+To enable storing of source references in ``project.refs``, add the
+following to your ``project.conf``:
+
+.. code:: yaml
+
+  ref-storage: project.refs
+
+.. note::
+
+   The ``ref-storage`` configuration is available since :ref:`format version 5 <project_format_version>`
+
+
+Fail on Overlaps
+~~~~~~~~~~~~~~~~
+When multiple elements are staged, there's a possibility that different
+elements will try and stage different versions of the same file.
+
+When ``fail-on-overlap`` is true, if an overlap is detected
+that hasn't been allowed by the element's
+:ref:`overlap whitelist<public_overlap_whitelist>`,
+then an error will be raised and the build will fail.
+
+otherwise, a warning will be raised indicating which files had overlaps,
+and the order that the elements were overlapped.
+
+.. code:: yaml
+
+  fail-on-overlap: true
+
+
+Source Aliases
+~~~~~~~~~~~~~~
+In order to abstract the download location of source code and
+any assets which need to be downloaded, and also as a matter of
+convenience, BuildStream allows one to create named aliases for
+URLs which are to be used in the individual ``.bst`` files.
+
+.. code:: yaml
+
+   aliases:
+     foo: git://git.foo.org/
+     bar: http://bar.com/downloads/
+
+
+Sandbox options
+~~~~~~~~~~~~~~~
+Sandbox options for the whole project can be supplied in
+``project.conf`` in the same way as in an element. See :ref:`element configuration <format_sandbox>`
+for more detail.
+
+.. code:: yaml
+
+   # Specify a user id and group id to use in the build sandbox.
+   sandbox:
+     build-uid: 1003
+     build-gid: 1001
+
+.. note::
+
+   The ``sandbox`` configuration is available since :ref:`format version 6 <project_format_version>`
+
+
+.. _project_essentials_artifacts:
+
+Artifact Server
+~~~~~~~~~~~~~~~
+If you have setup an :ref:`artifact server <artifacts>` for your
+project then it is convenient to configure this in your ``project.conf``
+so that users need not have any additional configuration to communicate
+with an artifact share.
+
+.. code:: yaml
+
+  artifacts:
+
+    # A url from which to download prebuilt artifacts
+    url: https://foo.com/artifacts
+
+You can also specify a list of caches here; earlier entries in the list
+will have higher priority than later ones.
+
+
+.. _project_plugins:
+
+External Plugins
+----------------
+If your project makes use of any custom :mod:`Element <buildstream.element>` or
+:mod:`Source <buildstream.source>` plugins, then the project must inform BuildStream
+of the plugins it means to make use of and the origin from which it can be loaded.
+
+Note that plugins with the same name from different origins are not permitted.
+
+
+Core plugins
+~~~~~~~~~~~~
+Plugins provided by the BuildStream core need not be explicitly specified
+here, but you may use this section to specify a minimal format version
+to ensure that they provide the features which your project requires.
+
+.. code:: yaml
+
+   plugins:
+   - origin: core
+
+     # We require a new feature of the `git` source plugin, and
+     # a new feature introduced in version 2 of the `patch` plugin.
+     sources:
+       git: 1
+       patch: 2
+
+     # ... And a new feature of the `script` element, added
+     # in version 2 of it's own format version.
+     elements:
+       script: 2
+
+
+Local Plugins
+~~~~~~~~~~~~~
+Local plugins are expected to be found in a subdirectory of the actual
+BuildStream project. :mod:`Element <buildstream.element>` and
+:mod:`Source <buildstream.source>` plugins should be stored in separate
+directories to avoid namespace collisions.
+
+The versions of local plugins are largely immaterial since they are
+revisioned along with the project by the user, usually in a VCS like git.
+However, for the sake of consistency with other plugin loading origins
+we require that you specify a version, this can always be ``0`` for a local
+plugin.
+
+
+.. code:: yaml
+
+   plugins:
+
+   - origin: local
+     path: plugins/sources
+
+     # We want to use the `mysource` source plugin located in our
+     # project's `plugins/sources` subdirectory.
+     sources:
+       mysource: 0
+
+
+Pip Plugins
+~~~~~~~~~~~
+Plugins loaded from the ``pip`` origin are expected to be installed
+separately on the host operating system using python's package management
+system.
+
+.. code:: yaml
+
+   plugins:
+
+   - origin: pip
+
+     # Specify the name of the python package containing
+     # the plugins we want to load. The name one would use
+     # on the `pip install` command line.
+     #
+     package-name: potato
+
+     # We again must specify a minimal format version for the
+     # external plugin, it is allowed to be `0`.
+     #
+     elements:
+       potato: 0
+
+
+.. _project_options:
+
+Options
+-------
+Options are how BuildStream projects can define parameters which
+can be configured by users invoking BuildStream to build your project.
+
+Options are declared in the ``project.conf`` in the main ``options``
+dictionary.
+
+.. code:: yaml
+
+   options:
+     debug:
+       type: bool
+       description: Whether to enable debugging
+       default: False
+
+Users can configure those options when invoking BuildStream with the
+``--option`` argument::
+
+    $ bst --option debug True ...
+
+
+Common Properties
+~~~~~~~~~~~~~~~~~
+All option types accept the following common attributes
+
+* ``type``
+
+  Indicates the type of option to declare
+
+* ``description``
+
+  A description of the meaning of the option
+
+* ``variable``
+
+  Optionally indicate a :ref:`variable <format_variables>` name to
+  export the option to. A string form of the selected option will
+  be used to set the exported value.
+
+  If used, this value will override any existing value for the
+  variable declared in ``project.conf``, and will be overridden in
+  the regular :ref:`composition order <format_composition>`.
+
+
+Boolean
+~~~~~~~
+The ``bool`` option type allows specifying boolean values which
+can be cased in conditional expressions.
+
+
+**Declaring**
+
+.. code:: yaml
+
+   options:
+     debug:
+       type: bool
+       description: Whether to enable debugging
+       default: False
+
+
+**Evaluating**
+
+Boolean options can be tested in expressions with equality tests:
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - debug == True:
+         enable-debug: True
+
+Or simply treated as truthy values:
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - debug:
+         enable-debug: True
+
+
+**Exporting**
+
+When exporting boolean options as variables, a ``True`` option value
+will be exported as ``1`` and a ``False`` option as ``0``
+
+
+Enumeration
+~~~~~~~~~~~
+The ``enum`` option type allows specifying a string value
+with a restricted set of possible values.
+
+
+**Declaring**
+
+.. code:: yaml
+
+   options:
+     loglevel:
+       type: enum
+       description: The logging level
+       values:
+       - debug
+       - info
+       - warning
+       default: info
+
+
+**Evaluating**
+
+Enumeration options must be tested as strings in conditional
+expressions:
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - loglevel == "debug":
+         enable-debug: True
+
+
+**Exporting**
+
+When exporting enumeration options as variables, the value is
+exported as a variable directly, as it is a simple string.
+
+
+Flags
+~~~~~
+The ``flags`` option type allows specifying a list of string
+values with a restricted set of possible values.
+
+In contrast with the ``enum`` option type, the *default* value
+need not be specified and will default to an empty set.
+
+
+**Declaring**
+
+.. code:: yaml
+
+   options:
+     logmask:
+       type: flags
+       description: The logging mask
+       values:
+       - debug
+       - info
+       - warning
+       default:
+       - info
+
+
+**Evaluating**
+
+Options of type ``flags`` can be tested in conditional expressions using
+a pythonic *in* syntax to test if an element is present in a set:
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - ("debug" in logmask):
+         enable-debug: True
+
+
+**Exporting**
+
+When exporting flags options as variables, the value is
+exported as a comma separated list of selected value strings.
+
+
+Architecture
+~~~~~~~~~~~~
+The ``arch`` option type is special enumeration option which
+defaults to the result of `uname -m`, and does not support
+assigning any default in the project configuration.
+
+.. code:: yaml
+
+   options:
+     machine_arch:
+       type: arch
+       description: The machine architecture
+       values:
+       - arm
+       - aarch64
+       - i386
+       - x86_64
+
+
+Architecture options can be tested with the same expressions
+as other Enumeration options.
+
+
+Element Mask
+~~~~~~~~~~~~
+The ``element-mask`` option type is a special Flags option
+which automatically allows only element names as values.
+
+.. code:: yaml
+
+   options:
+     debug_elements:
+       type: element-mask
+       description: The elements to build in debug mode
+
+This can be convenient for automatically declaring an option
+which might apply to any element, and can be tested with the
+same syntax as other Flag options.
+
+
+.. code:: yaml
+
+   variables:
+     enable-debug: False
+     (?):
+     - ("element.bst" in debug_elements):
+         enable-debug: True
+
+
+
+.. _project_defaults:
+
+Element Default Configuration
+-----------------------------
+The ``project.conf`` plays a role in defining elements by
+providing default values and also by overriding values declared
+by plugins on a plugin wide basis.
+
+See the :ref:`composition <format_composition>` documentation for
+more detail on how elements are composed.
+
+
+.. _project_defaults_variables:
+
+Variables
+~~~~~~~~~
+The defaults for :ref:`Variables <format_variables>` used in your
+project is defined here.
+
+.. code:: yaml
+
+   variables:
+     prefix: "/usr"
+
+
+.. _project_defaults_environment:
+
+Environment
+~~~~~~~~~~~
+The defaults environment for the build sandbox is defined here.
+
+.. code:: yaml
+
+   environment:
+     PATH: /usr/bin:/bin:/usr/sbin:/sbin
+
+Additionally, the special ``environment-nocache`` list which specifies
+which environment variables do not effect build output, and are thus
+not considered in the calculation of artifact keys can be defined here.
+
+.. code:: yaml
+
+   environment-nocache:
+   - MAXJOBS
+
+Note that the ``environment-nocache`` list only exists so that we can
+control parameters such as ``make -j ${MAXJOBS}``, allowing us to control
+the number of jobs for a given build without effecting the resulting
+cache key.
+
+
+Split Rules
+~~~~~~~~~~~
+The project wide :ref:`split rules <public_split_rules>` defaults can
+be specified here.
+
+.. code:: yaml
+
+   split-rules:
+     devel:
+     - |
+       %{includedir}
+     - |
+       %{includedir}/**
+     - |
+       %{libdir}/lib*.a
+     - |
+       %{libdir}/lib*.la
+
+
+.. _project_overrides:
+
+Overriding Plugin Defaults
+--------------------------
+Base attributes declared by element and source plugins can be overridden
+on a project wide basis. This section explains how to make project wide
+statements which augment the configuration of an element or source plugin.
+
+
+.. _project_element_overrides:
+
+Element Overrides
+~~~~~~~~~~~~~~~~~
+The elements dictionary can be used to override variables, environments
+or plugin specific configuration data as shown below.
+
+
+.. code:: yaml
+
+   elements:
+
+     # Override default values for all autotools elements
+     autotools:
+
+       variables:
+         bindir: "%{prefix}/bin"
+
+       config:
+         configure-commands: ...
+
+       environment:
+         PKG_CONFIG_PATH=%{libdir}/pkgconfig
+
+
+.. _project_source_overrides:
+
+Source Overrides
+~~~~~~~~~~~~~~~~
+The sources dictionary can be used to override source plugin
+specific configuration data as shown below.
+
+
+.. code:: yaml
+
+   sources:
+
+     # Override default values for all git sources
+     git:
+
+       config:
+         checkout-submodules: False
+
+.. note::
+
+   The ``sources`` override is available since :ref:`format version 1 <project_format_version>`
+
+
+.. _project_shell:
+
+Customizing the shell
+---------------------
+Since BuildStream cannot know intimate details about your host or about
+the nature of the runtime and software that you are building, the shell
+environment for debugging and testing applications may need some help.
+
+The ``shell`` section allows some customization of the shell environment.
+
+.. note::
+
+   The ``shell`` section is available since :ref:`format version 1 <project_format_version>`
+
+
+Interactive Shell Command
+~~~~~~~~~~~~~~~~~~~~~~~~~
+By default, BuildStream will use ``sh -i`` when running an interactive
+shell, unless a specific command is given to the ``bst shell`` command.
+
+BuildStream will automatically set a convenient prompt via the ``PS1``
+environment variable for interactive shells; which might be overwritten
+depending on the shell you use in your runtime.
+
+If you are using ``bash``, we recommend the following configuration to
+ensure that the customized prompt is not overwritten:
+
+.. code:: yaml
+
+   shell:
+
+     # Specify the command to run by default for interactive shells
+     command: [ 'bash', '--noprofile', '--norc', '-i' ]
+
+
+Environment Assignments
+~~~~~~~~~~~~~~~~~~~~~~~
+In order to cooperate with your host environment, a debugging shell
+sometimes needs to be configured with some extra knowledge inheriting
+from your host environment.
+
+This can be achieved by setting up the shell ``environment`` configuration,
+which is expressed as a dictionary very similar to the
+:ref:`default environment <project_defaults_environment>`, except that it
+supports host side environment variable expansion in values.
+
+.. note::
+
+   The ``environment`` configuration is available since :ref:`format version 4 <project_format_version>`
+
+For example, to share your host ``DISPLAY`` and ``DBUS_SESSION_BUS_ADDRESS``
+environments with debugging shells for your project, specify the following:
+
+.. code:: yaml
+
+   shell:
+
+     # Share some environment variables from the host environment
+     environment:
+       DISPLAY: '$DISPLAY'
+       DBUS_SESSION_BUS_ADDRESS: '$DBUS_SESSION_BUS_ADDRESS'
+
+Or, a more complex example is how one might share the host pulseaudio
+server with a ``bst shell`` environment:
+
+.. code:: yaml
+
+   shell:
+
+     # Set some environment variables explicitly
+     environment:
+       PULSE_SERVER: 'unix:${XDG_RUNTIME_DIR}/pulse/native'
+
+
+Host Files
+~~~~~~~~~~
+It can be useful to share some files on the host with a shell so that
+it can integrate better with the host environment.
+
+The ``host-files`` configuration allows one to specify files and
+directories on the host to be bind mounted into the sandbox.
+
+.. note::
+
+   The ``host-files`` configuration is available since :ref:`format version 4 <project_format_version>`
+
+.. warning::
+
+   One should never mount directories where one expects to
+   find data and files which belong to the user, such as ``/home``
+   on POSIX platforms.
+
+   This is because the unsuspecting user may corrupt their own
+   files accidentally as a result. Instead users can use the
+   ``--mount`` option of ``bst shell`` to mount data into the shell.
+
+
+The ``host-files`` configuration is an ordered list of *mount specifications*.
+
+Members of the list can be *fully specified* as a dictionary, or a simple
+string can be used if only the defaults are required.
+
+The fully specified dictionary has the following members:
+
+* ``path``
+
+  The path inside the sandbox. This is the only mandatory
+  member of the mount specification.
+
+* ``host_path``
+
+  The host path to mount at ``path`` in the sandbox. This
+  will default to ``path`` if left unspecified.
+
+* ``optional``
+
+  Whether the mount should be considered optional. This
+  is ``False`` by default.
+
+
+Here is an example of a *fully specified mount specification*:
+
+.. code:: yaml
+
+   shell:
+
+     # Mount an arbitrary resolv.conf from the host to
+     # /etc/resolv.conf in the sandbox, and avoid any
+     # warnings if the host resolv.conf doesnt exist.
+     host-files:
+     - host_path: '/usr/local/work/etc/resolv.conf'
+       path: '/etc/resolv.conf'
+       optional: True
+
+Here is an example of using *shorthand mount specifications*:
+
+.. code:: yaml
+
+   shell:
+
+     # Specify a list of files to mount in the sandbox
+     # directory from the host.
+     #
+     # If these do not exist on the host, a warning will
+     # be issued but the shell will still be launched.
+     host-files:
+     - '/etc/passwd'
+     - '/etc/group'
+     - '/etc/resolv.conf'
+
+Host side environment variable expansion is also supported:
+
+.. code:: yaml
+
+   shell:
+
+     # Mount a host side pulseaudio server socket into
+     # the shell environment at the same location.
+     host-files:
+     - '${XDG_RUNTIME_DIR}/pulse/native'
+
+
+.. _project_builtin_defaults:
+
+Builtin Defaults
+----------------
+BuildStream defines some default values for convenience, the default
+values overridden by your project's ``project.conf`` are presented here:
+
+  .. literalinclude:: ../../../../buildstream/data/projectconfig.yaml
+     :language: yaml