doc: Document multiple cache support

3 files changed, 79 insertions, 20 deletions

diff --git a/doc/source/artifacts.rst b/doc/source/artifacts.rst
index 7a5d3c0e5..acad7e1aa 100644
--- a/doc/source/artifacts.rst
+++ b/doc/source/artifacts.rst
@@ -5,20 +5,56 @@
 
 Artifact Caches
 ===============
-BuildStream revisions output of each element under it's specific
-cache key in a local artifact cache.
+BuildStream caches the results of builds in a local artifact cache, and will
+avoid building an element if there is a suitable build already present in the
+local artifact cache.
 
-This artifact cache can however be shared with others, or automated
-builders can be made to contribute to a shared artifact cache so
-that developers dont need to build everything all the time, instead
-they can download prebuilt artifacts from a shared cache, if an artifact
-is available for the specific cache keys they need.
+In addition to the local artifact cache, you can configure one or more remote
+artifact caches and BuildStream will then try to pull a suitable build from one
+of the remotes, falling back to a local build if needed.
 
-This page outlines how to setup and use a shared artifact cache.
+Configuring BuildStream to use remote caches
+--------------------------------------------
+A project will often set up continuous build infrastructure that pushes
+built artifacts to a shared cache, so developers working on the project can
+make use of these pre-built artifacts instead of having to each build the whole
+project locally. The project can declare this cache in its
+:ref:`project configuration file <project_essentials_artifacts>`.
 
+Users can declare additional remote caches in the :ref:`user configuration
+<config_artifacts>`. There are several use cases for this: your project may not
+define its own cache, it may be useful to have a local mirror of its cache, or
+you may have a reason to share artifacts privately.
+
+Remote artifact caches are identified by their URL. There are currently three
+supported protocols:
+
+* ``http``: Pull-only access, without transport-layer security
+* ``https``: Pull-only access, with transport-layer security
+* ``ssh``: Push access, authenticated via SSH
+
+BuildStream allows you to configure as many caches as you like, and will query
+them in a specific order:
+
+1. Project-specific overrides in the user config
+2. Project configuration
+3. User configuration
+
+When an artifact is built locally, BuildStream will try to push it to all the
+caches which have the ``push: true`` flag set. You can also manually push
+artifacts to a specific cache using the :ref:`bst pull command <invoking>`.
+
+Artifacts are identified using the element's :ref:`cache key <cachekeys>` so
+the builds provided by a cache should be interchangable with those provided
+by any other cache.
+
+
+Setting up a remote artifact cache
+----------------------------------
+The rest of this page outlines how to set up a shared artifact cache.
 
 Setting up the user
--------------------
+~~~~~~~~~~~~~~~~~~~
 A specific user is not needed for downloading artifacts, but since we
 are going to use ssh to upload the artifacts, you will want a dedicated
 user to own the artifact cache.
@@ -29,7 +65,7 @@ user to own the artifact cache.
 
 
 Installing the receiver
------------------------
+~~~~~~~~~~~~~~~~~~~~~~~
 You will also need to install BuildStream on the artifact server in order
 to receive uploaded artifacts over ssh. Follow the instructions for installing
 BuildStream :ref:`here <installing>`
@@ -52,7 +88,7 @@ requiring BuildStream's more exigent dependencies by setting the
 
 
 Initializing the cache
-----------------------
+~~~~~~~~~~~~~~~~~~~~~~
 Now that you have a dedicated user to own the artifact cache, change
 to that user, and create the artifact cache ostree repository directly
 in it's home directory as such:
@@ -65,7 +101,7 @@ This should result in an artifact cache residing at the path ``/home/artifacts/a
 
 
 Serve the cache over https
---------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 This part should be pretty simple, you can do this with various technologies, all
 we really require is that you make the artifacts available over https (you can use
 http but until we figure out using gpg signed ostree commits for the artifacts, it's
@@ -103,7 +139,7 @@ is the case for hosting anything over https.
 
 
 Configure and run sshd
-----------------------
+~~~~~~~~~~~~~~~~~~~~~~
 You will need to run the sshd service to allow uploading artifacts.
 
 For this you will want something like the following in your ``/etc/ssh/sshd_config``
@@ -129,7 +165,7 @@ For this you will want something like the following in your ``/etc/ssh/sshd_conf
 
 
 Summary file updates
---------------------
+~~~~~~~~~~~~~~~~~~~~
 BuildStream uses the OSTree summary file to determine what artifacts are
 available in the remote artifact cache. ``ostree summary -u`` updates
 the summary file. This command cannot be run concurrently and thus it
@@ -146,7 +182,7 @@ E.g., create ``/etc/cron.d/artifacts`` with the following content:
 
 
 User Configuration
-------------------
+~~~~~~~~~~~~~~~~~~
 The user configuration for artifacts is documented with the rest
 of the :ref:`user configuration documentation <config>`.
 
@@ -165,10 +201,11 @@ then a user can use the following user configuration:
 
      # Alternative form if you have push access to the cache
      #url: ssh://artifacts@artifacts.com:22200/artifacts
+     #push: true
 
 
 Authenticating Users
---------------------
+~~~~~~~~~~~~~~~~~~~~
 In order to give permission to a given user to upload
 artifacts, simply use the regular ``ssh`` method.
 
diff --git a/doc/source/config.rst b/doc/source/config.rst
index a5ceb8762..8a8035d11 100644
--- a/doc/source/config.rst
+++ b/doc/source/config.rst
@@ -29,12 +29,25 @@ Project Specific Value
 The ``projects`` key can be used to specify project specific configurations,
 the supported configurations on a project wide basis are listed here.
 
+.. _config_artifacts:
 
 Artifact Server
 ~~~~~~~~~~~~~~~
-The artifact server is usually specified by the project you build, but
-it can be overridden on a per project basis using the same format
-:ref:`described here <project_essentials_artifacts>`.
+The project you build will often specify a :ref:`remote artifact cache
+<artifacts>` already, but you may want to specify extra caches. There are two
+ways to do this.  You can add one or more global caches:
+
+**Example**
+
+.. code:: yaml
+
+   artifacts:
+     url: https://artifacts.com/artifacts
+
+Caches listed there will be considered lower priority than those specified
+by the project configuration.
+
+You can also add project-specific caches:
 
 **Example**
 
@@ -43,8 +56,15 @@ it can be overridden on a per project basis using the same format
    projects:
      project-name:
        artifacts:
-         url: https://artifacts.com/artifacts
+         - url: https://artifacts.com/artifacts1
+         - url: ssh://user@artifacts.com/artifacts2
+           push: true
+
+Caches listed here will be considered higher priority than those specified
+by the project.
 
+If you give a list of URLs, earlier entries in the list will have higher
+priority than later ones.
 
 Strict Build Plan
 ~~~~~~~~~~~~~~~~~
diff --git a/doc/source/projectconf.rst b/doc/source/projectconf.rst
index cffe61179..d7f609b39 100644
--- a/doc/source/projectconf.rst
+++ b/doc/source/projectconf.rst
@@ -102,6 +102,8 @@ with an artifact share.
     # A url from which to download prebuilt artifacts
     url: https://foo.com/artifacts
 
+You can also specify a list of caches here; earlier entries in the list
+will have higher priority than later ones.
 
 .. _project_plugins: